Artificial immunohistochemical image systems and methods

ABSTRACT

The disclosure provides a method of generating an artificial immunohistochemistry (IHC) image of cells. The method includes receiving a hematoxylin and eosin (H&amp;E) stained whole slide image (WSI) generated by a brightfield microscopy imaging modality of at least a portion of cells included in a specimen, applying, to the H&amp;E brightfield image, at least one trained model, the trained model being trained to generate the artificial IHC image based on the H&amp;E brightfield image, receiving the artificial IHC image from the trained model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/301,975, filed Apr. 20, 2021, which claims the benefit of U.S.Application 63/012,885, filed Apr. 20, 2020. Each application isincorporated herein by reference in its entirety.

BACKGROUND

Patient-derived tumor organoid (TO) technologies have been used tocreate cellular models of diverse cancer types, including colon, breast,pancreatic, liver, lung, endometrial, prostate, and esophagogastric,among others. In addition to advancing fundamental research, TOs haverecently been employed for drug development and precision medicinestudies.

Tumor organoids can be used to model cancer growth and estimate theeffectiveness of different therapies in stopping cancer growth. Tomonitor the growth of tumor organoids before, during, and after exposureto various anti-cancer therapies, the tumor organoids can be imaged todetect cell death and/or viable cells in a cell culture plate.

Some methods for detecting dead cells or viable cells can include theuse of a fluorescent signal, which can be detected by fluorescentmicroscopy. Fluorescent dyes can be applied to the tumor organoids inorder to highlight certain characteristics in the cells and/or make thecharacteristics easier to detect. The cells can then be imaged using atechnique such as fluorescent microscopy. However, fluorescentmicroscopy can be very time consuming, and the fluorescent dyes used canbe toxic to cells, which can artificially inflate the amount of observedcell death that may be falsely attributed to the anti-cancer therapybeing tested.

Accordingly, there is a need in the art to automatically analyze tumororganoids and other cellular compositions without the use of fluorescentdyes and/or fluorescent microscopy.

SUMMARY OF DISCLOSURE

Disclosed herein are systems, methods, and mechanisms useful forautomatically analyzing tumor organoid and other cellular compositionimages. In particular, the disclosure provides systems, methods, andmechanisms for generating images of cellular compositions, such as tumororganoids, that approximate fluorescent staining techniques using onlyraw brightfield images of tumor organoids.

In accordance with some embodiments of the disclosed subject matter, amethod of generating an artificial fluorescent image of cells isprovided. The method includes receiving a brightfield image generated bya brightfield microscopy imaging modality of at least a portion of cellsincluded in a specimen, applying, to the brightfield image, at least onetrained model, the trained model being trained to generate theartificial fluorescent image based on the brightfield image, receivingthe artificial fluorescent image from the trained model.

In accordance with some embodiments of the disclosed subject matter, anorganoid analysis system including at least one processor and at leastone memory is provided. The system is configured to receive abrightfield image generated by a brightfield microscopy imaging modalityfrom at least a portion of cells included in a specimen, apply, to thebrightfield image, at least one model trained to generate an artificialfluorescent image based on the brightfield image, the artificialfluorescent image being indicative of whether the cells included in thetumor organoids are alive or dead, and output the artificial fluorescentimage to at least one of a memory or a display.

In accordance with some embodiments of the disclosed subject matter, amethod of generating an artificial fluorescent image without afluorescent stain is provided. The method includes receiving abrightfield image generated by a brightfield microscopy imaging modalityfrom at least a portion of cells included in a specimen, applying, tothe brightfield image, at least one model trained to generate anartificial fluorescent image based on the brightfield image, theartificial fluorescent image being indicative of whether the cellsincluded in the tumor organoids are alive or dead, and generating areport based on the artificial fluorescent image.

BRIEF DESCRIPTION OF DRAWINGS

Petition for color: N/A

FIG. 1 shows an example of a system for automatically analyzing tumororganoid images.

FIG. 2 shows an example of hardware that can be used in some embodimentsof the system.

FIG. 3 shows an exemplary flow that can generate brightfield imagesand/or fluorescent images, as well as live/dead assays readouts, usingpatient derived organoids grown from tumor specimens.

FIG. 4 shows an exemplary flow for training a generator to generate anartificial fluorescent image based on an input brightfield image oforganoid cells.

FIG. 5 shows an exemplary flow for generating an artificial fluorescentimage.

FIG. 6 shows an exemplary neural network.

FIG. 7 shows an exemplary discriminator.

FIG. 8 shows an exemplary process that can train a model to generate anartificial fluorescent stain image of one or more organoids based on aninput brightfield image.

FIG. 9 shows an exemplary process that can generate an artificialfluorescent image of one or more organoids based on a brightfield image.

FIG. 10 shows exemplary raw images before preprocessing and afterpreprocessing.

FIG. 11 shows an exemplary flow for culturing tumor organoids. Cultureof patient derived tumor organoids.

FIG. 12 shows an exemplary flow for conducting drug screens inaccordance with systems and methods described herein.

FIG. 13 shows an exemplary process that can generate artificialfluorescent images at multiple time points for at least one organoid.

FIG. 14 shows a table representing an exemplary assay or well platearrangement.

FIG. 15 shows an example of images generated using a single neuralnetwork model and a three neural network model.

FIG. 16 shows a flow for generating an artificial fluorescent imageusing a first trained model and a second trained model.

FIG. 17 shows a process for generating fluorescent images of tumororganoids.

FIG. 18 shows a flow for predicting a viability based on a brightfieldimage.

FIG. 19 shows an exemplary generator and an exemplary discriminator.

FIG. 20 shows a discriminator that can generate a viability predictionbased on a brightfield image and an artificial fluorescent image.

FIG. 21 shows a process for generating a viability value.

DETAILED DESCRIPTION

The various aspects of the subject disclosure are now described withreference to the drawings. It should be understood, however, that thedrawings and detailed description hereafter relating thereto are notintended to limit the claimed subject matter to the particular formdisclosed. Rather, the intention is to cover all modifications,equivalents, and alternatives falling within the spirit and scope of theclaimed subject matter.

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and in which is shown byway of illustration, specific embodiments in which the disclosure may bepracticed. These embodiments are described in sufficient detail toenable those of ordinary skill in the art to practice the disclosure. Itshould be understood, however, that the detailed description and thespecific examples, while indicating examples of embodiments of thedisclosure, are given by way of illustration only and not by way oflimitation. From this disclosure, various substitutions, modifications,additions, rearrangements, or combinations thereof within the scope ofthe disclosure may be made and will become apparent to those of ordinaryskill in the art.

In accordance with common practice, the various features illustrated inthe drawings may not be drawn to scale. The illustrations presentedherein are not meant to be actual views of any particular method,device, or system, but are merely idealized representations that areemployed to describe various embodiments of the disclosure. Accordingly,the dimensions of the various features may be arbitrarily expanded orreduced for clarity. In addition, some of the drawings may be simplifiedfor clarity. Thus, the drawings may not depict all of the components ofa given apparatus (e.g., device) or method. In addition, like referencenumerals may be used to denote like features throughout thespecification and figures.

Information and signals described herein may be represented using any ofa variety of different technologies and techniques. For example, data,instructions, commands, information, signals, bits, symbols, and chipsthat may be referenced throughout the above description may berepresented by voltages, currents, electromagnetic waves, magneticfields or particles, optical fields or particles, or any combinationthereof. Some drawings may illustrate signals as a single signal forclarity of presentation and description. It will be understood by aperson of ordinary skill in the art that the signal may represent a busof signals, wherein the bus may have a variety of bit widths and thedisclosure may be implemented on any number of data signals including asingle data signal.

The various illustrative logical blocks, modules, circuits, andalgorithm acts described in connection with embodiments disclosed hereinmay be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and acts are described generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the embodiments of the disclosure describedherein.

In addition, it is noted that the embodiments may be described in termsof a process that is depicted as a flowchart, a flow diagram, astructure diagram, or a block diagram. Although a flowchart may describeoperational acts as a sequential process, many of these acts can beperformed in another sequence, in parallel, or substantiallyconcurrently. In addition, the order of the acts may be re-arranged. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, etc. Furthermore, the methods disclosed hereinmay be implemented in hardware, software, or both. If implemented insoftware, the functions may be stored or transmitted as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes both computer storage media and communication mediaincluding any medium that facilitates transfer of a computer programfrom one place to another.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not limit thequantity or order of those elements, unless such limitation isexplicitly stated. Rather, these designations may be used herein as aconvenient method of distinguishing between two or more elements orinstances of an element. Thus, a reference to first and second elementsdoes not mean that only two elements may be employed there or that thefirst element must precede the second element in some manner. Also,unless stated otherwise a set of elements may comprise one or moreelements.

As used herein, the terms “component,” “system” and the like areintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on a computer and the computercan be a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers or processors.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs.

Furthermore, the disclosed subject matter may be implemented as asystem, method, apparatus, or article of manufacture using standardprogramming and/or engineering techniques to produce software, firmware,hardware, or any combination thereof to control a computer or processorbased device to implement aspects detailed herein. The term “article ofmanufacture” (or alternatively, “computer program product”) as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips, etc.), opticaldisks (e.g., compact disk (CD), digital versatile disk (DVD), etc.),smart cards, and flash memory devices (e.g., card, stick).

Additionally, it should be appreciated that a carrier wave can beemployed to carry computer-readable electronic data such as those usedin transmitting and receiving electronic mail or in accessing a networksuch as the Internet or a local area network (LAN). Of course, thoseskilled in the art will recognize many modifications may be made to thisconfiguration without departing from the scope or spirit of the claimedsubject matter.

As used herein the terms “biological specimen,” “patient sample,” and“sample” refer to a specimen collected from a patient. Such samplesinclude, without limitation, tumors, biopsies, tumor organoids, othertissues, and bodily fluids. Suitable bodily fluids include, for example,blood, serum, plasma, sputum, lavage fluid, cerebrospinal fluid, urine,semen, sweat, tears, saliva, and the like. Samples may be collected, forexample, via a biopsy, swab, or smear.

The terms “extracted”, “recovered,” “isolated,” and “separated,” referto a compound, (e.g., a protein, cell, nucleic acid or amino acid) thathas been removed from at least one component with which it is naturallyassociated and found in nature.

The terms “enriched” or “enrichment” herein refer to the process ofamplifying nucleic acids contained in a sample. Enrichment can besequence specific or nonspecific (i.e., involving any of the nucleicacids present in a sample).

As used herein, “cancer” shall be taken to mean any one or more of awide range of benign or malignant tumors, including those that arecapable of invasive growth and metastases through a human or animal bodyor a part thereof, such as, for example, via the lymphatic system and/orthe blood stream. As used herein, the term “tumor” includes both benignand malignant tumors and solid growths. Typical cancers include but arenot limited to carcinomas, lymphomas, or sarcomas, such as, for example,ovarian cancer, colon cancer, breast cancer, pancreatic cancer, lungcancer, prostate cancer, urinary tract cancer, uterine cancer, acutelymphatic leukemia, Hodgkin's disease, small cell carcinoma of the lung,melanoma, neuroblastoma, glioma, and soft tissue sarcoma of humans.

Fluorescence microscopy is commonly used to detect the presence ofspecific molecules in a sample. For example, in cell biology,fluorescence microscopy can be used to highlight a specific cellularcomponent (e.g., an organelle) or detect a molecular marker that isindicative of a particular cellular state (e.g., apoptosis,differentiation, or activation of a cell signaling pathway). However,there are several drawbacks that limit the use of fluorescencemicroscopy. First, use of this technique requires additional time,labor, and reagents (e.g., stains) as compared to transmitted lightmicroscopy, making it a costly bottleneck in high-throughput screeningprocesses. Second, some fluorescent dyes are toxic to cells and can biasthe results of certain experiments (e.g., quantification of cell death).Further, cells that were damaged by these dyes can no longer be used inan ongoing experiment, so a greater quantity of cells is required forexperiments that involve assaying cells at multiple time points. Third,the time over which a sample can be observed using fluorescencemicroscopy is limited by photobleaching, a process in which fluorophoreslose their ability to fluoresce as they are illuminated.

Fortunately, methods based on transmitted light microscopy largely avoidthese problems, as they are relatively fast and inexpensive to use andcan capture multiple images of the same living samples at several timepoints. The term “transmitted light microscopy” is used to refer to anytype of microscopy where the light passes from the source to theopposite side of the lens. The simplest of these methods is brightfieldmicroscopy, in which samples are illuminated from below with white lightand the transmitted light is observed from above. Use of a standardbrightfield microscope is somewhat limited for biological samples thathave low contrast. For instance, without the use of stains, the membraneand nucleus are the only features of a mammalian cell that arediscernable in a brightfield image. Fortunately, adding opticalaccessories to a standard brightfield microscope can dramaticallyenhance image contrast, eliminating the need to kill, fix, and stainsamples. One very simple contrast-enhancing method is dark-fieldmicroscopy, which works by illuminating the sample with light that willnot be collected by the objective lens. For applications in whichgreater detail is required, phase-contrast microscopy and differentialinterference contrast microscopy may be employed. These complementarytechniques produce high-contrast images of transparent biologicalsamples by using optical systems to convert variations in density orthickness within the sample to differences in contrast in the finalimage. Importantly, these techniques can be used to reveal smallcellular structures, such as nuclei, ribosomes, mitochondria, membranes,spindles, mitotic apparatus, nucleolus, chromosomes, Golgi apparatus,vacuoles, pinocytotic vesicles, lipid droplets, and cytoplasmicgranules. Brightfield microscopy can also be augmented with polarizedlight, which creates contrast in samples comprising materials withdifferent refractive indices (i.e., birefringent samples). Whereasdark-field, phase-contrast, and differential interference contrastmicroscopy are well suited for imaging live, unstained biologicalsamples, polarized light microscopy is well suited for studying thestructure and composition of rocks, minerals, and metals. Notably, anyof these contrast-enhancing methods can be combined with opticalsectioning techniques, such as confocal microscopy and light sheetmicroscopy, which produce clear images of focal planes deep withinthicker samples (e.g., thick tissues, small organisms), reducing oreliminating the need to physically section samples (e.g., using amicrotome).

In the present application, the inventors demonstrate that certaincellular states that are commonly detected using fluorescence microscopyalso manifest as subtle morphological features in images produced bytransmitted light microscopy. While such features may be difficult orimpossible to discern in these images using only the human eye, theinventors show their identification can be automated using a trainedmodel. In the Examples, a trained model is used to predict thepercentages of live and dead cells in a sample (i.e., values which wouldtypically be determined using fluorescent stains, such as Caspase-3/7and TO-PRO-3 stain), using only a brightfield image as input. Thesevisualization methods may then be used in a high-throughput screen fordrugs that kill cancer cells within tumor organoids generated frompatient tumor samples.

The methods and systems disclosed herein are not limited to this singleapplication, however. The ability to associate subtle morphologicalfeatures that are present in a transmitted light microscopy image withcellular states of interest is useful in countless applications spanningmany diverse fields of study. Several exemplary, non-limitingapplications are discussed below.

The systems and methods disclosed herein have utility in the field ofbiology, as the disclosed systems and methods can be used tocharacterize samples ranging from individual cells (e.g., plant cells,animal cells), to tissue slices (e.g., biopsies), to small organisms(e.g., protozoa, bacteria, fungi, embryos, nematodes, insects).Importantly, by avoiding the use of cytotoxic stains, the disclosedsystems and methods allow the same samples to be imaged repeatedly overa multi-day or even multi-week time course. Images of the samples arecaptured using transmitted light microscopy, and a trained systemutilizes morphological characteristics (e.g., cell volume, diameter,shape, and topography) to identify cells that possess a certain cellularstate. For instance, one can estimate the numbers or concentrations ofparticular cell types present in a sample by training a system todistinguish cells by type, or assess cell viability by training a systemto distinguish between live and dead cells. Trained systems may also beused to characterize cells based on behaviors such as proliferation,differentiation, apoptosis, necrosis, motility, migration, cytoskeletaldynamics, cell-cell and cell-matrix adhesion, signaling, polarity, andvesicle trafficking. For example, the systems and methods disclosedherein may be used to differentiate between different modes of celldeath based on their unique morphologies (e.g., shrinkage in apoptosisversus swelling in necrosis). These systems and methods may also be usedto monitor the response of cells to any experimental manipulation,ranging from a culture condition to the effect of the outer spaceenvironment on biological processes. Thus, the disclosed systems andmethods provide a means to investigate myriad aspects of biology using ahighly efficient platform. While the cells, tissues, or organisms may beleft untreated (e.g., not subject to staining, fixing, etc.), thesystems and methods disclosed herein are also useful to image stained,fixed, or otherwise treated samples. By way of example but not by way oflimitation, tissues or cells that are immunohistochemically stained,hematoxylin and eosin stained, etc. may be imaged pursuant to thesystems and methods disclosed herein.

The systems and methods of the present disclosure are useful in thedevelopment of novel and improved therapeutics. For instance, thedisclosed systems and methods may be used to monitor the response ofcells to potential drugs in high-throughput drug screens, as describedin the Examples. Additionally, the disclosed systems and methods may beused to monitor the differentiation status of cells, both in the contextof development and in the directed differentiation of stem cells. Stemcells may be used to repair tissues damaged by disease or injury, eitherby directly injecting them into a patient or by differentiating theminto replacement cells ex vivo. For example, stem cells may bedifferentiated into a particular blood cell type for use in donor-freeblood transfusions. Other promising stem cell-based therapies includethe replacement of: bone marrow cells in blood cancer patients; neuronsdamaged by spinal cord injuries, stroke, Alzheimer's disease, orParkinson's disease; cartilage damaged by arthritis; skin damaged byserious burns; and islet cells destroyed by type 1 diabetes. Stem cellscan also be used to generate specific cell types, tissues, 3D tissuemodels, or organoids for use in drug screening. Use of a trained systemcapable of monitoring cell differentiation status would allow for moreefficient production of any of these stem cell-based products.

The systems and methods of the present disclosure are useful in thediagnosis of medical conditions. For example, the systems and methodsdisclosed herein can be used to quickly, efficiently, and accuratelydetect the presence of particular cell types in a patient sample thatare indicative of a disease or condition, e.g., tumor cells, blood inurine or stool, clue cells in vaginal discharge, or inflammatory cellinfiltration. For example, a system trained on images of tissue samples(e.g., biopsies) will detect morphological features that can be used todistinguish between benign, non-invasive, and invasive cancer cells.Additionally, such systems may be used to identify microbes andparasites in a patient samples, enabling the diagnosis of a wide rangeof infectious diseases including those caused by bacteria (e.g.,tuberculosis, urinary tract infection, tetanus, Lyme disease, gonorrhea,syphilis), fungi (e.g., thrush, yeast infections, ringworm), andparasites (e.g., malaria, sleeping sickness, hookworm disease, scabies).Such methods may be particularly useful for identifying the responsiblepathogen in cases where a condition may be caused by a variety ofmicrobes (e.g., infectious keratitis of the cornea).

The systems and methods disclosed herein can be used to identifyorganisms in environmental samples, such as soil, crops, and water. Thisapplication could be utilized in both the environmental sciences, e.g.,for assessing the health of an ecosystem based on the number anddiversity of organisms, and in epidemiology, e.g., for tracing thespread of contaminants that pose a health risk.

The systems and methods disclosed herein can be used to evaluate of awide variety materials, such as clays, fats, oils, soaps, paints,pigments, foods, drugs, glass, latex, polymer blends, textiles and otherfibers, chemical compounds, crystals, rocks and minerals. Applicationsin which such materials are analyzed using microscopy are found acrossdiverse fields. In industry, the systems and methods disclosed hereincan be used in failure analysis, design validation, and quality controlof commercial products and building materials. For example, the systemsand methods disclosed herein can be used to detect defects or fracturesin parts of machinery that require a high degree of precision, such aswatches and aircraft engines. In computer science, the systems andmethods disclosed herein can be used to examine integrated circuits andsemiconductors. In both archeology and forensics, the systems andmethods disclosed herein can be used to identify unknown materials andexamine wear patterns on artifacts/evidence. In geology, the systems andmethods disclosed herein can be used to determine the composition ofrocks and minerals, and to uncover evidence as to how they were formed.In agriculture, the systems and methods disclosed herein can be used todetect microbial indicators of soil health and to inspect seeds andgrains to assess their purity, quality, and germination capacity. Infood science, the systems and methods disclosed herein can be used toproduce in vitro cultured meat from animal cells.

The present application provides a non-limiting exemplary system thatuses brightfield images as input in a screen for cancer drugs.Typically, drug response is measured via cell viability assays usinglive/dead fluorescent stains, which have multiple drawbacks, which arediscussed above. While the use of brightfield microscopy largely avoidsthese issues, visualizing and quantifying live/dead cells frombrightfield images alone is not easily accessible and is a significantobstacle towards more cost-efficient high-throughput screening of tumororganoids. Certain systems and methods described herein provideartificial fluorescent images that can be generated using onlybrightfield images.

In some embodiments, a method of generating an artificial image of acellular composition such as a cell or a group of cells (e.g., cells inculture), is provided. In some embodiments, the generated image isindicative of whether the cell comprises one or more characteristicsindicative of a particular cell state or cell identity (e.g., death,disease, differentiation, strain of bacteria, etc.). In someembodiments, the methods include receiving a brightfield image;providing the brightfield image to a trained model; receiving theartificial fluorescent image from the trained model; and outputting theartificial fluorescent image to at least one of a memory or a display.In some embodiments, the cell is a mammalian cell, a plant cell, aeukaryotic cell, or a bacterial cell. In some embodiments, thecharacteristic(s) indicative of a cell state or cell identity compriseone or more distinguishing physical, structural features of the cell,wherein the features are identifiable by brightfield microscopy.Exemplary, non-limiting features include size, morphology, structureswithin the cell, staining values, structures on the cell surface, etc.

Drug Screening

Analysis of drug response data by target may identify importantpathways/mutations. For example, drugs can be applied to organoidsand/or specimens, and the results of the drug application can beanalyzed. For drugs that cause cell death in organoids, the targets ofthose drugs may be important. Thus, it is desirable to discover and/ordevelop additional drugs that modulate these targets. The cellularpathways and/or mutations that are important may be specific to thecancer type of the organoid. For example, if CDK inhibitors specificallykill colorectal cancer (CRC) tumor organoid cells, CDK may be especiallyimportant in CRC.

FIG. 1 shows an example of a system 100 for automatically analyzingtumor organoid images. In some embodiments, the system 100 can include acomputing device 104, a secondary computing device 108, and/or a display116. In some embodiments, the system 100 can include an organoid imagedatabase 120, a training data database 124, and/or a trained modelsdatabase 128. In some embodiments, the trained models database 128 caninclude one or more trained machine learning models such as artificialneural networks. In some embodiments, the computing device 104 can be incommunication with the secondary computing device 108, the display 116,the organoid image database 120, the training data database 124, and/orthe trained models database 128 over a communication network 112. Asshown in FIG. 1 , the computing device 104 can receive tumor organoidimages, such as brightfield images of tumor organoids, and generateartificial fluorescent stain images of the tumor organoids. In someembodiments, the computing device 104 can execute at least a portion ofan organoid image analysis application 132 to automatically generate theartificial fluorescent stain images.

The organoid image analysis application 132 can be included in thesecondary computing device 108 that can be included in the system 100and/or on the computing device 104. The computing device 104 can be incommunication with the secondary computing device 108. The computingdevice 104 and/or the secondary computing device 108 may also be incommunication with a display 116 that can be included in the system 100over the communication network 112.

The communication network 112 can facilitate communication between thecomputing device 104 and the secondary computing device 108. In someembodiments, communication network 112 can be any suitable communicationnetwork or combination of communication networks. For example,communication network 112 can include a Wi-Fi network (which can includeone or more wireless routers, one or more switches, etc.), apeer-to-peer network (e.g., a Bluetooth network), a cellular network(e.g., a 3G network, a 4G network, a 5G network, etc., complying withany suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX,etc.), a wired network, etc. In some embodiments, communication network112 can be a local area network, a wide area network, a public network(e.g., the Internet), a private or semi-private network (e.g., acorporate or university intranet), any other suitable type of network,or any suitable combination of networks. Communications links shown inFIG. 1 can each be any suitable communications link or combination ofcommunications links, such as wired links, fiber optic links, Wi-Filinks, Bluetooth links, cellular links, etc.

The organoid image database 120 can include a number of raw tumororganoid images, such as brightfield images. In some embodiments, thebrightfield images can be generated using a brightfield microscopyimaging modality. Exemplary brightfield images are described below. Insome embodiments, the organoid image database 120 can include artificialfluorescent stain images generated by the organoid image analysisapplication 132.

The training data database 124 can include a number of images fortraining a model to generate artificial fluorescent stain images. Insome embodiments, the training data image database 124 can include rawbrightfield images and corresponding three channel fluorescent stainimages. The trained models database 128 can include a number of trainedmodels that can receive raw brightfield images of tumor organoids andoutput artificial fluorescent stain images. In some embodiments, trainedmodels 136 can be stored in the computing device 104. In someembodiments, each pair of the raw brightfield images and thecorresponding three channel fluorescent stain images can be include acommon field of view of the same slide captured by different microscopesettings for the brightfield images and the corresponding three channelfluorescent stain images, respectively (e.g., brightfield settings andfluorescent settings).

In some embodiments, the training data database 124 can include paired(corresponding) histopathology slide images, where each image depicts atissue slice from a biological specimen or a blood smear from a blooddraw. In some embodiments, if two images correspond it can indicate thatthe tissue slice(s) or blood smear(s) associated with the two images areobtained from the same biological specimen. For example, the two imagesmay be obtained from the same tumor biopsy or the same blood draw. Insome embodiments, the images can depict tissue slices that may have beenapproximately adjacent in the specimen and/or the same tissue slice mayhave been used to generate both images. In some embodiments, thecorresponding images can depict corresponding cellular and/or tissuestructures. For example, both images can depict common structures (e.g.,different sections of the same biological cell, same organ, same tissue,etc.). In one example, one of the images can include hematoxylin andeosin (H&E) staining and the other image can includeimmunohistochemistry (IHC) staining. The IHC staining may be multiplexIHC staining. An example of corresponding images can be found in U.S.patent application Ser. No. 16/830,186, filed Mar. 25, 2020 and titled“Determining Biomarkers From Histopathology Slide Images,” which isincorporated herein by reference in its entirety. In variousembodiments, one advantage of simulating IHC slides is to facilitate thedetection of various biomarkers without the cost of IHC staining.

FIG. 2 shows an example 200 of hardware that can be used in someembodiments of the system 100. The computing device 104 can include aprocessor 204, a display 208, an input 212, a communication system 216,and a memory 220. The processor 204 can be any suitable hardwareprocessor or combination of processors, such as a central processingunit (“CPU”), a graphics processing unit (“GPU”), etc., which canexecute a program, which can include the processes described below.

In some embodiments, the display 208 can present a graphical userinterface. In some embodiments, the display 208 can be implemented usingany suitable display devices, such as a computer monitor, a touchscreen,a television, etc. In some embodiments, the inputs 212 of the computingdevice 104 can include indicators, sensors, actuatable buttons, akeyboard, a mouse, a graphical user interface, a touch-screen display,etc.

In some embodiments, the communication system 216 can include anysuitable hardware, firmware, and/or software for communicating with theother systems, over any suitable communication networks. For example,the communication system 216 can include one or more transceivers, oneor more communication chips and/or chip sets, etc. In a more particularexample, communication system 216 can include hardware, firmware, and/orsoftware that can be used to establish a coaxial connection, a fiberoptic connection, an Ethernet connection, a USB connection, a Wi-Ficonnection, a Bluetooth connection, a cellular connection, etc. In someembodiments, the communication system 216 allows the computing device104 to communicate with the secondary computing device 108.

In some embodiments, the memory 220 can include any suitable storagedevice or devices that can be used to store instructions, values, etc.,that can be used, for example, by the processor 204 to present contentusing display 208, to communicate with the secondary computing device108 via communications system(s) 216, etc. The memory 220 can includeany suitable volatile memory, non-volatile memory, storage, or anysuitable combination thereof. For example, the memory 220 can includeRAM, ROM, EEPROM, one or more flash drives, one or more hard disks, oneor more solid state drives, one or more optical drives, etc. In someembodiments, the memory 220 can have encoded thereon a computer programfor controlling operation of computing device 104 (or secondarycomputing device 108). In such embodiments, the processor 204 canexecute at least a portion of the computer program to present content(e.g., user interfaces, images, graphics, tables, reports, etc.),receive content from the secondary computing device 108, transmitinformation to the secondary computing device 108, etc.

The secondary computing device 108 can include a processor 224, adisplay 228, an input 232, a communication system 236, and a memory 240.The processor 224 can be any suitable hardware processor or combinationof processors, such as a central processing unit (“CPU”), a graphicsprocessing unit (“GPU”), etc., which can execute a program, which caninclude the processes described below.

In some embodiments, the display 228 can present a graphical userinterface. In some embodiments, the display 228 can be implemented usingany suitable display devices, such as a computer monitor, a touchscreen,a television, etc. In some embodiments, the inputs 232 of the secondarycomputing device 108 can include indicators, sensors, actuatablebuttons, a keyboard, a mouse, a graphical user interface, a touch-screendisplay, etc.

In some embodiments, the communication system 236 can include anysuitable hardware, firmware, and/or software for communicating with theother systems, over any suitable communication networks. For example,the communication system 236 can include one or more transceivers, oneor more communication chips and/or chip sets, etc. In a more particularexample, communication system 236 can include hardware, firmware, and/orsoftware that can be used to establish a coaxial connection, a fiberoptic connection, an Ethernet connection, a USB connection, a Wi-Ficonnection, a Bluetooth connection, a cellular connection, etc. In someembodiments, the communication system 236 allows the secondary computingdevice 108 to communicate with the computing device 104.

In some embodiments, the memory 240 can include any suitable storagedevice or devices that can be used to store instructions, values, etc.,that can be used, for example, by the processor 224 to present contentusing display 228, to communicate with the computing device 104 viacommunications system(s) 236, etc. The memory 240 can include anysuitable volatile memory, non-volatile memory, storage, or any suitablecombination thereof. For example, the memory 240 can include RAM, ROM,EEPROM, one or more flash drives, one or more hard disks, one or moresolid state drives, one or more optical drives, etc. In someembodiments, the memory 240 can have encoded thereon a computer programfor controlling operation of secondary computing device 108 (orcomputing device 104). In such embodiments, the processor 224 canexecute at least a portion of the computer program to present content(e.g., user interfaces, images, graphics, tables, reports, etc.),receive content from the computing device 104, transmit information tothe computing device 104, etc.

The display 116 can be a computer display, a television monitor, aprojector, or other suitable displays.

FIG. 3 shows an exemplary flow 300 that can generate brightfield imagesand/or fluorescent images, as well as live/dead assays readouts, usingpatient derived organoids grown from tumor specimens. In someembodiments, the live/dead assays readouts can be produced usingbrightfield and multiplexed fluorescence imaging. Drug response can bemeasured via cell viability assays using live/dead fluorescent stains.In some embodiments, the flow 300 can be included in a high throughputdrug screening system. An example of a high throughput drug screeningcan be found in U.S. Prov. patent application Ser. No. 17/114,386,titled “Large Scale Organoid Analysis” and filed Dec. 7, 2020, which isincorporated herein by reference in its entirety. In some examples,biological therapies, such as antibodies or allogenic therapies, may beused as one or more of the drugs in the drug screening.

The flow 300 can include harvesting a tumor specimen 308 from a humanpatient 304, culturing organoids 312 using the tumor specimen 308, drugscreening 316 the organoids, imaging the organoids 320, and outputtingbrightfield and fluorescence images 324 of the organoids. After theorganoids are cultured, cells from the organoids can be plated into anassay plate (e.g. a 96-well assay plate, a 384-well assay plate, etc.).The assay plate may also be referred to as a plate. The drug screening316 can include plating the cells and treating the cells with a numberof different drugs and/or concentrations. For example, a 384-well platecan include fourteen drugs at seven different concentrations. As anotherexample, a 96-well plate can include six drugs at five differentconcentrations. The imaging 320 can include brightfield imaging thetreated cells, as well as applying fluorescent stains to at least aportion of the cells and fluorescent imaging the cells. In someembodiments, the fluorescent imaging can include producing threechannels of data for each cell. The three channels of data can include ablue/all nuclei channel, a green/apoptotic channel, and a red/pink/deadchannel. Each channel can be used to form a fluorescent image.Additionally, the imaging 320 can produce combined 3-channel fluorescentimages that include the blue/all nuclei channel, the green/apoptoticchannel, and the red/pink/dead channel. In some embodiments, the imaging320 can include generating brightfield images of the cells using abright-field microscope and generating fluorescent images of the cellsusing a confocal microscope such as a confocal laser scanningmicroscope. In some embodiments, instead of using traditionalfluorescent staining to generate the fluorescent images, the imaging 320can include generating brightfield images for at least a portion of thecells and generating artificial brightfield images for the portion ofthe cells based on the brightfield images using a process describedbelow (e.g., the process of FIG. 9 ).

By way of example but not by way of limitation, in some embodiments,brightfield images (for example a 2D brightfield projection) depicting acell culture well during a drug screening assay can be generated using abrightfield modality, such as a brightfield microscope. In someembodiments, a brightfield image is generated using a 10×objective on amicroscope. A different objective can be used if higher or lowermagnification is desired. In some embodiments, the microscope can be anImageXPRESS microscope available from Molecular Devices. Othermicroscope brands capable of brightfield imaging are commerciallyavailable and can also be employed in the disclosed methods. In someembodiments, the cells can be cancer cell lines or cancer tumororganoids derived from patient specimens.

FIG. 4 shows an exemplary flow 400 for training a generator 408 togenerate an artificial fluorescent image 412 based on an inputbrightfield image 404 of a cellular composition, such as organoid cells.In some embodiments, the generator 408 can include a U-Net convolutionalneural network. In some embodiments, the generator 408 can include apix2pix model. In some embodiments, the generator 408 can be agenerative adversarial network (GAN). An exemplary neural network thatcan be included in the generator 408 is described below in conjunctionwith FIG. 6 . In some embodiments, the generator can include a neuralnetwork that can receive the brightfield image 404 and output a singlethree-channel fluorescent image (e.g., a 256×256×3 image). In someembodiments, the generator can include three neural networks that caneach receive the brightfield image 404 and output a one-channelfluorescent image (e.g., a 256×256×1 image). Generators that includethree neural networks that can each receive the brightfield image 404and output a one-channel fluorescent image may be referred to asthree-model generators. Each of the neural networks can be trained tooutput a specific channel of fluorescence. For example, a first neuralnetwork can output a blue/all nuclei channel image, a second neuralnetwork can output a green/apoptotic channel image, and a third neuralnetwork can output a red/dead channel image. The flow 400 can includecombining the blue/all nuclei channel image, the green/apoptotic channelimage, and the red/dead channel image into a single three-channelfluorescent image (e.g., a 256×256×3 image, a 1024×1024×3 image, etc.).

The flow can include providing the brightfield image 404, the artificialfluorescent image 412, and a ground truth fluorescent image 424associated with brightfield image to a discriminator 416 that canpredict whether or not an image is real or generated by the generator408 (e.g., the artificial fluorescent image 412). In some embodiments,the generator 408 can receive an image and output a label ranging from 0to 1, with 0 indicating that image is generated by the generator 408 and1 indicating that the image is real (e.g., the ground truth fluorescentimage 424 associated with the brightfield image 404). In someembodiments, the discriminator 416 can be a PatchGAN discriminator, suchas a 1×1 PatchGAN discriminator. An exemplary discriminator is describedbelow in conjunction with FIG. 7 .

The flow 400 can include an objective function value calculation 420.The objective function value calculation 420 can include calculating anobjective function value based on labels output by the discriminator 416and/or by other metrics calculated based on the brightfield image 404,the artificial fluorescent image 412, and the ground truth fluorescentimage 424. The objective function value can capture multiple lossfunctions (e.g., a weighted sum of multiple loss functions). In thisway, the objective function value can act as a total loss value for thegenerator 408 and the discriminator 416. The flow 400 can includetransmitting the objective function value and/or other information fromthe discriminator 416 to the generator 408 and the discriminator 416 inorder to update both the generator 408 and the discriminator 416. Anumber of different suitable objective functions can be used tocalculate the objective function value. However, in testing anembodiment of the generator 408, a sum of GANLoss+0.83SSIM+0.17L1 wasshown to outperform other tested loss functions such as GANLoss+L1 asused by the generator 408. GANLoss can be used to determine whether animage is real or generated. The L1 loss can be used as an additionalobjective to be minimized to ensure that the generated and real imagehave the least mean absolute error in addition to GANLoss. StructuralSimilarity Index (SSIM) can be used to improve performance acrossmultiple performance metrics as well as reduce artifacts. The objectivefunction value calculation 420 will be described below.

The flow 400 can include receiving a number of pairs of a brightfieldimage and a corresponding ground truth fluorescence image, anditeratively training the generator 408 using each pair of images.

In some embodiments, the flow 400 can include receiving a number ofpairs of a H&E image and a corresponding ground truth IHC image, anditeratively training the generator 408 using each pair of images togenerate artificial IHC images.

In some embodiments, the flow 400 can include receiving a number ofpairs of an IHC image and a corresponding ground truth H&E image, anditeratively training the generator 408 using each pair of images togenerate artificial H&E images.

In some embodiments, the flow 400 can include receiving a number ofpairs of an IHC image and/or a multiplex IHC images and a correspondingground truth H&E image, and iteratively training the generator 408 usingeach pair of images to generate artificial H&E images.

In some embodiments, the flow 400 can include pre-processing thebrightfield image 404 and the ground truth fluorescent image 424. Rawbrightfield and fluorescent images may have minimal contrast and requireenhancement before being used to train the generator 408. For example,in testing, the pixel intensities for the individual channels of thefluorescent image were generally skewed to zero, which may have beenbecause most of the image is black (i.e., background), except forregions containing organoids and/or cells.

In some embodiments, the artificial fluorescent image 412 can be used toprovide a count of live/dead cells. In order to enhance the contrast ofthe artificial fluorescent image 412 and improve the ability to countlive/dead cells from the artificial fluorescent image 412, both thebrightfield image 404 and the corresponding ground truth image 424 canundergo contrast enhancement to brighten and sharpen organoids/cells.

In some embodiments, multiple brightfield images and multiple groundtruth fluorescent images can be generated per well. For example, for a96-well plate, there can be about 9-16 sites per well that get imaged.

In some embodiments, the raw brightfield and ground truth fluorescentimages can have pixel intensities ranging from [0, 2¹⁶]. First, acontrast enhancement process, which can be included in the organoidimage analysis application 132, can convert each image to an unsignedbyte format, with values ranging from [0, 255]. Next, the contrastenhancement process can stretch and clip each pixel intensity to adesired output range.

In some embodiments, the desired intensity range of an input image to bestretched can be decided on a per image basis as follows: For the threepixel intensities corresponding to the three fluorophores used togenerate the fluorescent image, the input range can be re-scaled usingthe mode of the pixel intensity distribution as the lower bound valueand 1/10th the maximum pixel intensity as the upper bound. The contrastenhancement process can choose the upper bound in order to avoidoversaturated pixels and focus on cell signal. The contrast enhancementprocess can normalize each pixel intensity based on the lower bound andthe upper bound, which function as a min/max range, using a min-maxnorm, and then each pixel can be multiplied by the output range [0,255].For the brightfield image 404, the contrast enhancement process candetermine an input range by uniformly stretching the 2nd and 98thpercentile of pixel intensities to the output range [0,255].

For images with low signal, background noise may be included in theoutput range. To minimize any remaining back-ground noise, the contrastenhancement process can clip the minimum pixel value by two integervalues for the red and green channels, and by three integer values forthe blue channel, where the intensity range is wider on average. Themaximum pixel values can be increased accordingly to preserve intensityrange per image.

In some embodiments, the ground truth image 424 can be a 1024×1024×3 RGBimage including a blue channel corresponding to nuclei (Hoecsht), agreen channel corresponding to apoptotic cells (Caspase), and a redchannel corresponding to dead cells (TO-PRO-3) In some embodiments, theflow 400 can include enhancing the ground truth image 424. In someembodiments, the enhancing can include contrast enhancing the bluechannel, the green channel, and the red channel to brighten and sharpenorganoids and/or cells in the ground truth image 424. In someembodiments, the flow can down convert pixel intensities in the groundtruth image 424 (e.g., converting sixteen bit pixel intensities to eightbit intensities). After converting pixel intensities, the flow 400 caninclude rescaling pixel intensities to 1/10th of a maximum pixelintensity as the upper bound, as well as to the mode of the pixelintensity+two integer values as the lower bound for the red channel andthe green channel, and to the mode of the pixel intensity+three integervalues for the blue channel as the lower bound.

In some embodiments, the discriminator 416 can output a predicted label(e.g., a “0” or a “1”) to the objective function calculation 420. Thepredicted label can indicate if the artificial fluorescent image 412 isfake or real. In some embodiments, the objective function can becalculated as a weighted sum of GANLoss, SSIM, and L1. In someembodiments, the GANLoss can be calculated based on the predicted labeloutput by the discriminator. The GANLoss can be used to determinewhether the artificial fluorescent image 412 is real or generated. Insome embodiments, the L1 loss can be calculated based on the artificialfluorescent image 412 and the corresponding ground truth image. The L1loss can be used as an additional objective to be minimized to ensurethat the artificial fluorescent image 412 and the corresponding groundtruth image have the least mean absolute error in addition to GANLoss.

Certain machine learning models, such as the pix2pix model, may only useGANLoss and L1 loss in training a generator. As mentioned above, theobjective function calculation 420 can include an SSIM metric inaddition to the GANLoss and the L1 loss, which can improve theperformance of the generator 408 in comparison to a generator trainedusing only GANLoss and L1 loss.

In some embodiments, the objective function implemented in the objectivefunction calculation can be defined as:

$\begin{matrix}{G^{*} = {{\arg\min\limits_{G}\max\limits_{D}{\mathcal{L}_{GAN}\left( {G,D} \right)}} + {\lambda{\mathcal{L}_{L1}(G)}} + {\beta\left( {1 - {\mathcal{L}_{SSIM}(G)}} \right)}}} & (1)\end{matrix}$

where λ+β=1, L_(L1) is the mean absolute error loss, and 1−L_(SSIM)(G)is the structural similarity index loss between the generated image G(e.g., the fluorescent image 412) and the corresponding ground truthimage. In some embodiments, λ can be 0.17 and β can be 0.83. In someembodiments, λ can be selected from 0.1 to 0.3, and β can be selectedfrom 0.7 to 0.9.

In some embodiments, SSIM can take into account the luminance (l),contrast (c), and structure (s) of two images and computes a metricbetween 0 and 1, where 1 indicates a perfect match between the twoimages:

$\begin{matrix}{{l\left( {x,y} \right)} = \frac{{2\mu_{x}\mu_{y}} + C_{1}}{\mu_{x}^{2} + \mu_{y}^{2} + C_{1}}} & (2)\end{matrix}$ $\begin{matrix}{{c\left( {x,y} \right)} = \frac{{2\sigma_{x}\sigma_{y}} + C_{2}}{\sigma_{x}^{2} + \sigma_{y}^{2} + C_{2}}} & (3)\end{matrix}$ $\begin{matrix}{{s\left( {x,y} \right)} = \frac{\sigma_{xy} + C_{3}}{{\sigma_{x}\sigma_{y}} + C_{3}}} & (4)\end{matrix}$

C₁, C₂ and C₃ are small constants defined by:

C ₁=(K ₁ L)² ,C ₂=(K ₂ L)² and C ₃ =C ₂/2  (5)

where K₁, K₂ are two scalar constants whose values are less than 1, andL is the dynamic range of the pixel intensities (i.e. 256). SSIM canthen be calculated as:

$\begin{matrix}{{{SSIM}\left( {x,y} \right)} = {\left\lbrack {l\left( {x,y} \right)} \right\rbrack^{\alpha} \cdot \left\lbrack {c\left( {x,y} \right)} \right\rbrack^{\gamma}}} & (6)\end{matrix}$ $\begin{matrix}{{{SSIM}\left( {x,y} \right)} = \frac{\left( {{2\mu_{x}\mu_{y}} + C_{1}} \right)\left( {{2\sigma_{xy}} + C_{2}} \right)}{\left( {\mu_{x}^{2} + \mu_{y}^{2} + C_{1}} \right)\left( {\sigma_{x}^{2} + \sigma_{y}^{2} + {C2}} \right)}} & (7)\end{matrix}$

where l, c, and s are computed using the mean, variance and covariancerespectively of two images of the same size using a fixed window size.α, β, and γ are constants set to 1. In addition to structuralsimilarity, we also evaluated model prediction using root mean squareerror, which is the sum of the squared difference of pixel intensities.

In some embodiments, the Proposed Loss function can be as follows:

DiscriminatorLoss=MSELoss{Real Prediction,1}+MSELoss{GeneratedPrediction,0}+MSELoss{Predicted Viability,Viability}

GeneratorLoss=MSELoss{Generated Prediction,1}+MAE{GeneratedFluorescent,Real Fluorescent}+SSIM{Generated Fluorescent,RealFluorescent};

where MSE refers to Mean Squared Error Loss, MAE is the mean absoluteerror, and SSIM is the Structural similarity index. The RCA Model wastrained for thirty epochs with a learning rate of 2e-3 and Adamoptimizer. The images of resolution 1024×1024 were imaged at 10×magnification. They were randomly flipped as a data augmentation step.

In some embodiments, once a dye is added to a cell culture well, thecells in that well cannot continue to be used for the experiment, suchthat it is difficult or impossible to measure cell death in that well ata subsequent point in time. In some embodiments, the flow 400 caninclude generating artificial fluorescent images, which can reduce timerequirements for imaging by a factor of ten in comparison to utilizingdyes to generate the fluorescent images. Standard fluorescent imagingmay take up to an hour to perform. In some embodiments, the flow 400 canbe used in conjunction with a drug screening platform that uniquelyinterprets tumor organoids (TOs) which have limited biomass andintra-tumoral clonal heterogeneity by incorporating Patient DerivedTumor Organoids. The platform couples high content fluorescent confocalimaging analysis with a robust statistical analytical approach tomeasure hundreds of discrete data points of TO viability from as few as10{circumflex over ( )}3 cells.

In some embodiments, a flow similar to flow 400 may be used to train agenerator (e.g., the generator 408) to receive H&E images and generateartificial IHC images and/or multiplex IHC images. In some embodiments,the flow 400 can include receiving an H&E image from training datadatabase 124 and providing the H&E image to the generator 408 (in placeof the brightfield image 404). In these embodiments, the discriminator424 can receive an artificial H&E image generated by the generator and aground truth IHC image (in place of the ground truth fluorescent image424) corresponding to an H&E image from training data database 124.Thus, the flow 400 can be used to train a generator to generateartificial IHC images and/or multiplex IHC images based on H&E images.

Referring to FIG. 4 as well as FIG. 5 , an exemplary flow 500 forgenerating an artificial fluorescent image 512 is shown. The flow 500can include providing an input brightfield image 504 of plated cells toa trained model 508. The trained model 508 can include the generator408, which can be trained using the flow 400. The trained model 508 canoutput an artificial fluorescent image 512. The fluorescent image 512can be used to generate a live/dead assays readout and/or analyze theeffectiveness of different drugs and/or dosages on cancer cells intissue organoids.

Notably, the flow 500 can produce the fluorescent image 512 without theuse of fluorescent dyes, which provides several advantages overtraditional fluorescent imaging processes that require the use offluorescent dyes. Some dyes have cytotoxicity and must be added acertain amount of time before imaging. Additionally, once certain dyesare added to a cell culture well, the cells in that well cannot continueto be used for reimaging because of the difficulty in measuring celldeath in that well at a subsequent point in time. Thus, the flow 500 canimprove the ease of generating the fluorescent images because the flow500 may only require brightfield imaging, which is not time-dependentlike the traditional fluorescent imaging. Additionally, the flow 500 canincrease the speed at which the fluorescent images are obtained, becausefluorescent dyes do not need to be applied to the cells, and because theflow 500 does not have to wait for the fluorescent dyes to diffusebefore imaging the cells. As another example, the flow 500 can allowmultiple fluorescent images to be generated for each cell well at anumber of different time points. The fluorescent dyes used intraditional fluorescent imaging can damage the cells enough to preventreimaging. In contrast, the flow 500 can be used to produce multiplefluorescent images over a time period of days, weeks, months, etc. Thus,the flow 500 can provide more data points per cell well than traditionalfluorescent imaging.

The training data used to train a trained model may be selected based onthe aspects of the cellular compositions to be imaged post-training. Insome embodiments, a single trained model (e.g., trained model 508) canbe trained on training data including a set of brightfield images andcorresponding fluorescent images associated with one of six or moreorganoid lines each having a distinct cancer type, such that eachorganoid line is represented in the training data. In some embodiments,a single trained model (e.g., trained model 508) can be a pan-cancermodel trained to generate artificial fluorescent stain images from abrightfield image associated with any cancer type. In some embodiments,a trained model can be trained on training data only including imagesassociated with one organoid line (for example, one cancer type).

FIG. 6 shows an exemplary neural network 600. The neural network 600 canbe trained to receive an input image 604 and generate an artificialfluorescent image 608 based on the input image 604. In some embodiments,the input image 604 can be a raw brightfield image that has beenprocessed to enhance contrast and/or modify other characteristics inorder to enhance the raw brightfield image and potentially produce abetter artificial fluorescent image (e.g., the fluorescent image 608).

In some embodiments, the neural network 600 can include a Unetarchitecture. In some embodiments, the Unet architecture can be sized toreceive a 256×256×3 input image. The 256×256×3 input image can be abrightfield image. In some embodiments, the input image can be a256×256×1 image. In some embodiments, the generator 408 in FIG. 4 and/orthe trained model 508 in FIG. 5 can include the neural network 600.

FIG. 7 shows an exemplary discriminator 700. In some embodiments, thediscriminator 700 in FIG. 7 can be included as the discriminator 416 inthe flow 400 shown in FIG. 4 . In some embodiments, the discriminator700 can be a 1×1 PatchGAN. In some embodiments, the discriminator 700can receive a brightfield image 704 and a fluorescent image 708. Thefluorescent image can be an artificial fluorescent image (e.g., thefluorescent image 608 in FIG. 6 ) or a ground truth fluorescent image.In some embodiments, each of the brightfield image 704 and thefluorescent image 708 can be 256×256×3 input images. In someembodiments, the brightfield image 704 and the fluorescent image 708 canbe concatenated. In some embodiments, the concatenated image can be a256×256×6 input image.

In some embodiments, the discriminator 700 can receive the brightfieldimage 704 and a fluorescent image 708 and generate a predicted label 712indicative of whether or not the fluorescent image 708 is real or fake.In some embodiments, the predicted label 712 can be a “0” to indicatethe fluorescent image 708 is fake, and “1” to indicate the fluorescentimage 708 is real. In some embodiments, the discriminator 700 caninclude a neural network

Referring to FIGS. 4-7 , in some embodiments, the flow 400, the flow500, the neural network 600, and the discriminator 700 can beimplemented using Pytorch version 1.0.0. In some embodiments, the flow400 can be used to train the generator 408 to generate artificialfluorescent images for a colon cancer organoid line. In someembodiments, the flow 400 can be used to train the generator 408 togenerate artificial fluorescent images for a gastric cancer organoidline.

FIG. 8 shows an exemplary process 800 that can train a model to generatean artificial fluorescent stain image of one or more organoids based onan input brightfield image. In some embodiments, the model can be thegenerator 408 in FIG. 4 , and/or the neural network 600. In someembodiments, the model can include a neural network that can receive theinput brightfield image and output a single three-channel fluorescentimage (e.g., a 256×256×3 image). In some embodiments, the model caninclude three neural networks that can each receive the brightfieldimage and output a one-channel fluorescent image (e.g., a 256×256×1image). The one-channel images can then be combined into a singlethree-channel fluorescent image.

In some embodiments, the process 800 can be used to train a model tooutput artificial fluorescent images of objects other than tumororganoids using a number of non-fluorescent images (e.g., brightfieldimages) and fluorescent stain images (which may have more or less thanthree channels) as training data.

The process 800 can be implemented as computer readable instructions onone or more memories or other non-transitory computer readable media,and executed by one or more processors in communication with the one ormore memories or other media. In some embodiments, the process 800 canbe implemented as computer readable instructions on the memory 220and/or the memory 240 and executed by the processor 204 and/or theprocessor 224.

At 804, the process 800 can receive training data. In some embodiments,the training data can include a number of brightfield images and anumber of associated real fluorescent images of organoids. In someembodiments, the organoids can be from a single tumor organoid line. Insome embodiments, the brightfield images and the real fluorescent imagescan be preprocessed in order to enhance contrast as described above. Insome embodiments, the brightfield images and the real fluorescent imagescan be raw images that have not undergone any preprocessing such ascontrast enhancement.

At 808, if the training data includes raw brightfield images and/or rawreal fluorescent images (i.e., “YES” at 808), the process 800 canproceed to 812. If the training data does not include any rawbrightfield images or raw real fluorescent images (i.e., “NO” at 808),the process 800 can proceed to 816.

At 812, the process 800 can preprocess at least a portion of thebrightfield images and/or real fluorescent images. In some embodiments,at 812, the process 800 can enhance the contrast of any raw brightfieldimages and/or real fluorescent images included in the training data. Insome embodiments, the raw brightfield and ground truth fluorescentimages can have pixel intensities ranging from [0,2¹⁶]. In someembodiments, the process 800 can convert each image to an unsigned byteformat, with values ranging from [0, 255]. The process 800 can thenstretch and clip each pixel intensity to a desired output range.

In some embodiments, the process 800 can stretch the desired intensityrange of the input on a per image basis. For the three pixel intensitiescorresponding to the three fluorophores used to generate a realfluorescent image, the process 800 can re-scale the input range usingthe mode of the pixel intensity distribution as the lower bound valueand 1/10th the maximum pixel intensity as the upper bound. The process800 can determine the upper bound in order to avoid oversaturated pixelsand focus on cell signal. The process 800 can normalize each pixelintensity based on the lower bound and the upper bound, which functionas a min/max range, using a min-max norm, and then each pixel can bemultiplied by the output range [0,255]. For each brightfield imageincluded in the training data, the process 800 can determine an inputrange by uniformly stretching the 2nd and 98th percentile of pixelintensities to the output range [0,255].

For images with low signal, background noise may be included in theoutput range. In some embodiments, to minimize any remaining backgroundnoise, the process 800 can clip the minimum pixel value by two integervalues for the red and green channels, and by three integer values forthe blue channel, where the intensity range is wider on average. In someembodiments, the process 800 can increase maximum pixel valuesaccordingly to preserve intensity range per image.

At 816, the process 800 can provide a brightfield image to the model. Asdescribed above, in some embodiments, the model can be the generator 408in FIG. 4 and/or the neural network 600 in FIG. 6 . In some embodiments,the model can include three neural networks, and each neural network canreceive a copy of the brightfield image and output a different channel(e.g., red, green, or blue) of an artificial fluorescent image.

At 820, the process 800 can receive an artificial fluorescent image fromthe model. The model can generate the artificial fluorescent image(e.g., the artificial fluorescent image 412) based on the brightfieldimage (e.g., the brightfield image 404) provided to the model. In someembodiments, the process 800 can receive three one-channel images fromthree neural networks included in the model and combine the one-channelimages into a single three-channel artificial fluorescent image.

At 824, the process 800 can calculate an objective function value basedon the brightfield image, the real fluorescent image associated with thebrightfield image, and the artificial fluorescent image. In someembodiments, the process 800 can determine a predicted label indicativeof whether or not the artificial fluorescent image is real or not byproviding the artificial fluorescent image and the real fluorescentimage to a discriminator (e.g., the discriminator 416). In someembodiments, the objective function value can be calculated usingequation (1) above, where λ is 0.17 and β is 0.83. In some embodiments,λ can be selected from 0.1 to 0.3, and β can be selected from 0.7 to0.9. In some embodiments, the learning rate can fixed at 0.0002 for afirst number of epochs (e.g., fifteen epochs) of training, and thenlinearly decayed to zero over a second number of epochs (e.g., tenepochs).

At 828, the process 800 can update the model (e.g., the generator 408)and the discriminator (e.g., the discriminator 416) based on theobjective function value. In some embodiments, the model and thediscriminator can each include a neural network. In some embodiments,the process 800 can update weights of layers included in neural networksincluded in the model and the discriminator based on the objectivefunction value.

At 832, the process 800 can determine whether or not there is abrightfield image included in the training data that has not beenprovided to the model. If there is a brightfield image included in thetraining data that has not been provided to the model (e.g., “YES” at832), the process can proceed to 816 in order to provide the brightfieldimage to the model. If there are no brightfield images included in thetraining data that has not been provided to the model (e.g., “NO” at832), the process can proceed to 836.

At 836, the process 800 can cause the model to be output. At 836, themodel has been trained, and can be referred to as a trained model. Insome embodiments, the process 800 can cause the trained model to beoutput to at least one of a memory (e.g., the memory 220 and/or thememory 240) and/or a database (e.g., the trained models database 128).The trained model may be accessed and used in certain processes, such asthe processes in FIGS. 9 and 13 . The process 800 can then end.

FIG. 9 shows an exemplary process 900 that can generate an artificialfluorescent image of one or more organoids based on a brightfield image.More specifically, the process 900 can generate the artificialfluorescent image using a trained model. In some embodiments, the modelcan be the generator 408 in FIG. 4 , the trained model 508, and/or theneural network 600 in FIG. 6 trained using the process 800. In someembodiments, the model can include a neural network that can receive theinput brightfield image and output a single three-channel fluorescentimage (e.g., a 256×256×3 image). In some embodiments, the model caninclude three neural networks that can each receive the brightfieldimage and output a one-channel fluorescent image (e.g., a 256×256×1image). The one-channel images can then be combined into a singlethree-channel fluorescent image.

In some embodiments, the process 900 can be used to generate artificialfluorescent images (which can have one channel, two channels, threechannels, etc.) of objects other than tumor organoids using anon-fluorescent image (e.g., a brightfield image). In this way, objectsother than tumor organoids that require fluorescent staining to beproperly imaged can be artificially generated without the use of and/ordrawbacks of fluorescent dyes.

The process 900 can be implemented as computer readable instructions onone or more memories or other non-transitory computer readable media,and executed by one or more processors in communication with the one ormore memories or other media. In some embodiments, the process 900 canbe implemented as computer readable instructions on the memory 220and/or the memory 240 and executed by the processor 204 and/or theprocessor 224. In some embodiments, the process 900 can be executed byan imaging system. In some embodiments, a brightfield microscopy imagingsystem can be configured to execute the process 900. In someembodiments, the brightfield microscopy imaging system can include oneor more memories or other non-transitory computer readable mediaincluding the process 900 implemented as computer readable instructionson the one or more memories or other non-transitory computer readablemedia, and one or more processors in communication with the one or morememories or other media configured to execute the computer readableinstructions to execute the process 900.

At 904, the process 900 can receive a brightfield image (e.g., thebrightfield image 404 in FIG. 4 and/or the brightfield image 504 in FIG.5 ) of one or more organoids. In some embodiments, the brightfield imagecan be preprocessed in order to enhance contrast as described above. Insome embodiments, the brightfield image can be a raw image that has notundergone any preprocessing such as contrast enhancement.

At 908, the process 900 can determine if the brightfield image isunprocessed (i.e., raw). If the brightfield image is unprocessed (i.e.,“YES” at 908), the process 900 can proceed to 912. If the brightfieldimage is not unprocessed (i.e., “NO” at 908), the process 900 canproceed to 916.

At 912, the process 900 can preprocess the brightfield image. In someembodiments, the brightfield image can have pixel intensities rangingfrom [0, 2¹⁶]. In some embodiments, the process 900 can convert thebrightfield image to an unsigned byte format, with values ranging from[0, 255]. In some embodiments, the process 900 can convert thebrightfield image to another format with less bits than the originalpixel intensity. The process 900 can then stretch and clip each pixelintensity to a desired output range. In some embodiments, the process900 can determine an input range for the brightfield image by uniformlystretching the 2nd and 98th percentile of pixel intensities in thebrightfield image to an output range [0,255].

At 916, the process 900 can provide the brightfield image to a trainedmodel. In some embodiments, the model can include the generator 408 inFIG. 4 trained using the process 800 in FIG. 8 , the trained model 508,and/or the neural network 600 trained using the process 800 in FIG. 8 .In some embodiments, the trained model can include three neuralnetworks, and each neural network can receive a copy of the brightfieldimage and output a different channel (e.g., red, green, or blue) of anartificial fluorescent image. In some embodiments, the process 900 canapply the trained model to the brightfield image to generate anartificial fluorescent image.

At 920, the process 900 can receive an artificial fluorescent image fromthe trained model. In some embodiments, the process 900 can receivethree one-channel images from three neural networks included in thetrained model and combine the one-channel images into a singlethree-channel artificial fluorescent image. The artificial fluorescentimage can indicate whether cells included in the tumor organoids arealive or dead.

At 924, the process 900 can cause the artificial fluorescent image to beoutput. In some embodiments, the process 900 can cause the artificialfluorescent image to be output to at least one of a memory (e.g., thememory 220 and/or the memory 240) and/or a display (e.g., the display116, the display 208, and/or the display 228). The artificialfluorescent image can be used to provide a live/dead count of cells inthe organoids. In some embodiments, the process 900 can cause theartificial fluorescent image to be output to an automatic cell countingprocess in order to receive an accurate live/dead count of cells, apercentage of cells that are viable (e.g., alive) or dead, and/or a cellcount report in the artificial fluorescent image. For example, theprocess 900 can cause the artificial fluorescent image to be output tothe CellProfiler available at https://cellprofiler.org. In someembodiments, the process 900 can cause one or more channels of theartificial fluorescent image to be output to an automatic cell countingprocess in order to receive a cell count report, a percentage of cellsthat are viable (e.g., alive) or dead, and/or accurate live/dead countof cells in the artificial fluorescent image. In some embodiments, theprocess 900 can cause the brightfield image to be output to a trainedmodel in order to receive a cell count report, a percentage of cellsthat are viable (e.g., alive) or dead, and/or accurate live/dead countof cells in the artificial fluorescent image. In some embodiments, theprocess 900 can cause a combination (e.g., image embeddings combined byconcatenation) of the brightfield image and one, two, or three channelsof the artificial fluorescent image to be output to an automatic cellcounting process in order to receive a cell count report, a percentageof cells that are viable (e.g., alive) or dead, and/or an accuratelive/dead count of cells in the artificial fluorescent image.

In some embodiments, at 924, the process 900 can identify cells in theartificial fluorescent image by converting each of the channels tograyscale, enhancing and suppressing certain features such as speckles,ring shapes, neurites, dark holes, identifying primary objects belongingto the all cell channel where the typical diameters of these objects (inpixel units) is set anywhere between 2 and 20 with a minimum crossentropy thresholding method at a smoothing scale of 1.3488, andidentifying primary objects again belonging to the dead cells channelwhere typical diameter is anywhere between 5 and 20 in pixel units. Inthis way, the process 900 can generate a cell count report. In someembodiments, the process 924 can determine if a drug and/or dosage iseffective in killing tumor organoid cells based on the live/dead countof cells. In some embodiments, at 924, the process 900 can extrapolatedose response from a distribution of organoid viability at a singleconcentration.

In some embodiments, the cell count report may be analyzed to quantifythe efficacy of the drug in killing a particular line of tumor organoidcells. For example, if a concentration of a drug causes a lower numberof live cells and/or greater number of dead cells, the drug may be ratedas more effective in killing a particular line of tumor organoid cells.For each line of tumor organoid cells, characteristics of the tumororganoid cells (for example, molecular data including detectedmutations, RNA expression profiles measured in the tumor organoid cellsetc., other biomarkers, and/or clinical data associated with the patientfrom which the tumor organoid was derived) and the results (includingthe drug efficacy rating) of each drug dose may be saved in a databaseof drug assay results. These results may be used to match therapies topatients. For example, if a patient has a cancer with characteristicssimilar to a tumor organoid cell line, drugs rated as effective inkilling those tumor organoid cells may be matched to the patient.

In some embodiments, the process 900 can analyze nucleic acid dataassociated with the one or more tumor organoids. Each tumor organoidincluded in the one or more tumor organoids can be associated with aspecimen (e.g., the specimen the tumor organoid was harvested from). Insome embodiments, each specimen can be associated with a patient. Thepatient can be associated with patient data that can include nucleicacid data. In some embodiments, the nucleic acid data can include wholeexome data, transcriptome data, DNA data, and/or RNA data. The nucleicacid data may be used to further analyze the patient. In someembodiments, the process 900 can associate the artificial fluorescentimage with information about the specimen (e.g., the nucleic acid data).In some embodiments, the process 900 can provide the artificialfluorescent image and the associated information about the specimen to adatabase. In some embodiments, the database can include at least sevenhundred and fifty artificial fluorescent images.

In some embodiments, the process 900 can generate a report based on thecell count, the cell count report, the nucleic acid data, and/or theartificial fluorescent image. In some embodiments, the process 900 cancause the report to be output to at least one of a memory (e.g., thememory 220 and/or the memory 240) and/or a display (e.g., the display116, the display 208, and/or the display 228). The process 900 can thenend.

FIG. 10 shows exemplary raw images before preprocessing and afterpreprocessing. The raw images before preprocessing include a brightfieldimage 1004, a blue/all nuclei channel fluorescent image 1008, agreen/apoptotic channel fluorescent image 1012, red/pink/dead channelfluorescent image 1016, and a combined 3-channel fluorescent image 1020.The preprocessed images include a brightfield image 1024, a blue/allnuclei channel fluorescent image 1028, a green/apoptotic channelfluorescent image 1032, red/pink/dead channel fluorescent image 1036,and a combined 3-channel fluorescent image 1040. The organoids and cellsare brighter and sharper in the preprocessed images. In someembodiments, the preprocessed images 1024-1040 can be generated at 812in the process 800 in FIG. 8 .

FIG. 11 shows an exemplary flow 1100 for culturing tumor organoids.Culture of patient derived tumor organoids. The flow 100 can includeobtaining tumor tissue from a same-day surgery, disassociating cellsfrom the tumor tissue, and culturing the tumor organoids from the cells.An example of systems and methods for culturing tumor organoids may befound in U.S. patent application Ser. No. 16/693,117, titled “TumorOrganoid Culture Compositions, Systems, and Methods” and filed Nov. 22,2019, which is incorporated by reference herein in its entirety. Tumortissue sent from hospitals is cultured to form tumor organoids.

FIG. 12 shows an exemplary flow 1200 for conducting drug screens inaccordance with systems and methods described herein. In someembodiments, the flow 1200 can include disassociating tumor organoidsinto single cells, plating the cells (e.g., in a well plate such as a96-well plate and/or a 384-well plate), growing the cells into organoidsover a predetermined time period (e.g., seventy-two hours), treating theorganoids with at least one therapeutic technique, and imaging the tumororganoids a predetermined amount of time (e.g., seventy-two hours) afterthe tumor organoids are treated. In some embodiments, only brightfieldimaging may be performed on the tumor organoids, and any brightfieldimages generated can be used to generate artificial fluorescent imagesusing the process 900 in FIG. 9 . A live/dead count can then begenerated based on the artificial fluorescent images. One example ofsystems and methods for using tumor organoids for drug screens may befound in U.S. Patent Prov. App. No. 62/924,621, titled “Systems andMethods for Predicting Therapeutic Sensitivity” and filed Oct. 22, 2019(and PCT/US20/56930, filed Oct. 22, 2020), which are incorporated byreference herein in their entireties.

FIG. 13 shows an exemplary process 1300 that can generate artificialfluorescent images at multiple time points for at least one organoid.Notably, the process 1300 can provide an advantage over standardfluorescent imaging techniques. As mentioned above, fluorescent dyesused to generate standard fluorescent images can damage the cells (e.g.,killing the cells) in the organoids, and do not permit fluorescentimages to be generated at different time points (e.g., every twelvehours, every twenty-four hours, every seventy-two hours, every week,etc.). In contrast, the process 1300 permits repeated fluorescentimaging of organoids because the process 1300 may only requirebrightfield images (which do not damage the organoids), and can generateartificial fluorescent images based on the brightfield images.

The process 1300 can be implemented as computer readable instructions onone or more memories or other non-transitory computer readable media,and executed by one or more processors in communication with the one ormore memories or other media. In some embodiments, the process 1300 canbe implemented as computer readable instructions on the memory 220and/or the memory 240 and executed by the processor 204 and/or theprocessor 224. In some embodiments, the process 1300 can be executed byan imaging system. In some embodiments, a brightfield microscopy imagingsystem can be configured to execute the process 1300. In someembodiments, the brightfield microscopy imaging system can include oneor more memories or other non-transitory computer readable mediaincluding the process 1300 implemented as computer readable instructionson the one or more memories or other non-transitory computer readablemedia, and one or more processors in communication with the one or morememories or other media configured to execute the computer readableinstructions to execute the process 1300.

At 1304, the process 1300 can receive an indication to analyze treatedorganoids at multiple time points. In some embodiments, the organoidscan be plated (e.g., in a well plate such as a 96-well plate and/or a384-well plate). In some embodiments, the organoids can be plated onmultiple well plates. In some embodiments, the organoids can be platedon one or more petri dishes. In some embodiments, the organoids can betreated using a variety of different treatments, which can vary in drugtype, drug concentration, and/or other parameters. In some embodiments,each well in a well plate can be associated with a different treatment.

In some embodiments, the multiple time points can represent a time afterthe organoids have been treated. For example, a twelve hour time pointcan be twelve hours after the time at which the organoids were treated.In some embodiments, the multiple time points can be spaced at regularintervals. For example, the multiple time points can occur every twelvehours, every twenty-four hours, every seventy-two hours, every week,etc. In some embodiments, the multiple time points can be irregularlyspaced. For example, the time points can include a first time point atsix hours, a second time point at twenty four-hours, a third time pointat three days, a fourth time point at one week, and a fifth time pointat twenty-eight days.

At 1308, the process 1300 can wait until the next time point included inthe multiple time points. For example, if six hours has passed since theorganoids have been treated, and the next time point is at twelve hours,the process 1300 can wait for six hours.

At 1312, the process 1300 can cause at least one brightfield image ofthe treated organoids to be generated. In some embodiments, process 1300can generate the brightfield images of the treated organoids using abright-field microscope and generating fluorescent images of the cellsusing a confocal microscope such as a confocal laser scanningmicroscope. In some embodiments, the process 1300 can preprocess the atleast one brightfield image. For example, the process 1300 can, for eachbrightfield image, perform at least a portion of 912 in the process 900in FIG. 9 . In some embodiments, multiple brightfield images can begenerated for each well. For example, for a 96-well plate, there can beabout 9-16 sites per well that get imaged.

At 1316, the process 1300 can cause at least one artificial fluorescentimage to be generated based on the at least one brightfield image. Insome embodiments, the process 1300 can provide each brightfield image toa trained model, and receive an artificial fluorescent image associatedwith the brightfield image from the trained model. In some embodiments,the trained model can include the generator 408 in FIG. 4 trained usingthe process 800 in FIG. 8 , the trained model 508, and/or the neuralnetwork 600 trained using the process 800 in FIG. 8 . In someembodiments, the trained model can include a neural network that canreceive the input brightfield image and output a single three-channelfluorescent image (e.g., a 256×256×3 image). In some embodiments, thetrained model can include three neural networks that can each receivethe brightfield image and output a one-channel fluorescent image (e.g.,a 256×256×1 image). The one-channel images can then be combined into asingle three-channel fluorescent image. The at least one artificialfluorescent image can indicate whether cells included in the tumororganoids are alive or dead. In some embodiments, the process 1300 canapply the trained model to the at least one brightfield image togenerate the at least one artificial fluorescent image.

At 1320, the process 1300 can cause the at least one fluorescent imageto be output. In some embodiments, the process 1300 can cause the atleast one artificial fluorescent image to be output to at least one of amemory (e.g., the memory 220 and/or the memory 240) and/or a display(e.g., the display 116, the display 208, and/or the display 228). The atleast one artificial fluorescent image can be used to provide alive/dead count of cells in the organoids. In some embodiments, theprocess 900 can cause the artificial fluorescent image to be output toan automatic cell counting process in order to get an accurate live/deadcount of cells in the artificial fluorescent image. For example, theprocess 900 can cause the artificial fluorescent image to be output tothe CellProfiler available at https://cellprofiler.org. In this way, theprocess 1300 can automatically generate live/dead counts for multiplewells at multiple time points, which can make drug treatment experimentsrun faster and gather more data with the same number of wells ascompared to standard fluorescent dye imaging techniques that kill cells.

In some embodiments, at 1320, the process 1300 can identify cells in theartificial fluorescent image by converting each of the channels tograyscale, enhancing and suppressing certain features such as speckles,ring shapes, neurites, dark holes, identifying primary objects belongingto the all cell channel where the typical diameters of these objects (inpixel units) is set anywhere between 2 and 20 with a minimum crossentropy thresholding method at a smoothing scale of 1.3488, andidentifying primary objects again belonging to the dead cells channelwhere typical diameter is anywhere between 5 and 20 in pixel units. Inthis way, the process 1300 can generate a cell count report.

In some embodiments, the process 1300 can analyze nucleic acid dataassociated with the one or more tumor organoids. Each tumor organoidincluded in the one or more tumor organoids can be associated with aspecimen (e.g., the specimen the tumor organoid was harvested from). Insome embodiments, each specimen can be associated with a patient. Thepatient can be associated with patient data that can include nucleicacid data. In some embodiments, the nucleic acid data can include wholeexome data, transcriptome data, DNA data, and/or RNA data. The nucleicacid data may be used to further analyze the patient. In someembodiments, the process 1300 can associate the artificial fluorescentimage with information about the specimen (e.g., the nucleic acid data).In some embodiments, the process 1300 can provide the artificialfluorescent image and the associated information about the specimen to adatabase. In some embodiments, the database can include at least sevenhundred and fifty artificial fluorescent images.

In some embodiments, the process 1300 can generate a report based on thecell count, the cell count report, the nucleic acid data, and/or theartificial fluorescent image. In some embodiments, the process 1300 cancause the report to be output to at least one of a memory (e.g., thememory 220 and/or the memory 240) and/or a display (e.g., the display116, the display 208, and/or the display 228). The process 1300 can thenend.

In some embodiments, the process 800 in FIG. 8 , the process 900 in FIG.9 , and/or the process 1300 in FIG. 13 can be included in the organoidsimage analysis application 132 in FIG. 1 .

FIG. 14 shows a table representing an exemplary assay or well platearrangement. More specifically, the table shows an arrangement oftreatment therapies by well in a 24×16 well plate.

In some embodiments, to populate the well plate with tumor organoids,single cell suspensions of tumor organoid cells can be generated using apredetermined protocol. In some embodiments, to populate a 24×16 wellplate, a 24-well plate culture can be dissociated into single cells andseeded in 384-well plates in a mix of 30% Matrigel and 70% media. Thissetup can allow tumor organoids to form from individual cells for theassay, maintaining tumor organoid heterogeneity in each well. About 2000cells can be seeded per well allowing enough tumor organoids (TO's) toform while not overcrowding the plate.

The number of usable wells in each 384-well plate can be 330 wells.There can be two sites in each well which get imaged. For a 96-wellplate, there can be about 9-16 sites per well that get imaged. In someembodiments, each row in the well plate can receive a different drug. Insome embodiments, a control (e.g., Staurosporine) can be fixed in row A.The vehicle can be column 2 where the first half is given DMSO and thesecond half Staurosporine. In some embodiments, each row can receivedrug concentration in technical triplicate.

Example 1

In this example, Tumor Organoids in each well were stained using threefluorophores for high content fluorescent confocal imaging analysis. Inorder to obtain the fluorescent readouts a high content imager(ImageXpress Confocal, Molecular Devices) was utilized for dataacquisition. Images were acquired at 10× magnification with a 50 micronslit spinning disk aperture. Four channels were acquired usingincandescent brightfield, and LED light sources using manufacturer'sdefault settings for 4′,6-diamidino-2-phenylindole (DAPI), Fluoresceinisothiocyanate (FITC), and Cyanine 5 (CY5) to acquire data from Hoechst33342 (Thermo), Caspase-3/7 reagent (Essen Bioscience), or TO-PRO-3(Thermo) respectively.

In this example, the experimental setup used a 384 well plate, with 330usable wells within each plate. Since each well has two sites that getimaged, each plate has a total of 660 paired brightfield andfluorescence images. At a magnification of 10×, two images are taken perwell at 2 sites with a stack of images in the Z plane ranging from 1-100heights with increments as high as 15 microns per z plane. The Z stackimages are projected to 2D for analysis. The three fluorophores for eachbrightfield image visualizes all nuclei (Hoechst 33342, blue), apoptoticcells (Caspase-3/7 Apoptosis Assay Reagent, green), and dead cells(TO-PRO-3, red).

The final dataset contained registered fluorophores from two patientlines. Patient line A with colon cancer consisted of 9900 pairedbrightfield and fluorescent images. Patient line B with gastric cancerconsisted of 10557 paired brightfield and fluorescent images.

A model in accordance with the generator 408 and the discriminator 416was implemented in Pytorch version 1.0.0. The colon cancer organoidpatient line A was selected to evaluate performance for all modelsanalyzed. The organoid line contained 8415 paired brightfield andfluorescent images across 15 experimental plates, which was subjected toan 80-10-10 split for training, test and validation, resulting in 6930images for training, 742 in validation and 743 for test. Each image wasloaded one at a time. Fine tuning of parameters was achieved using onlyvalidation set. The learning rate was fixed at 0.0002 for the first 15epochs of training and consequently linearly decayed to 0 for the next10 epochs yielding a total of 25 epochs for training.

A fixed set of 743 brightfield and corresponding fluorescent imagesrandomly sampled from 15 experimental plates was chosen as a test set.Evaluation for all experiments was performed on the fixed test set.First, the effect of training three separate models (three-model) wasevaluated for each fluorophore channel versus training one single model(one-model) on the combined fluorescent readout. For the three-model,predictions for each fluorophore were combined at the end and evaluated.Performance was evaluated both quantitatively using structuralsimilarity index and root mean squared error as well as qualitativelyvisually using heatmaps.

No significant improvement in fluorescent stain prediction was observedwhen using the three-model, which trained a separate generator for eachchannel. Table 1 reports the average SSIM and root mean squared erroracross each channel's predictions for all 743 test images, and FIG. 15shows example images and organoids. Furthermore, because the three-modelrequired three times as many computing resources with only limited RMSEimprovement, it was reasoned that a one-model implementation couldsufficiently and efficiently perform image-to-image translation frombrightfield image to a combined fluorescent readout. In Table 1, lowerRMSE and higher SSIM indicates better performance.

TABLE 1 Experiment Avg. RMSE Avg. SSIM three-model 1.3655 0.92299one-model 1.39952 0.92383

FIG. 15 shows an example of images generated using a single neuralnetwork model (one-model) and a three neural network model(three-model). A first image 1504 is an example of a ground truthfluorescence image. A second image 1508 is an example of an artificialfluorescent image generated using a one-model with a single neuralnetwork that can receive an input brightfield image and output a singlethree-channel fluorescent image (e.g., a 256×256×3 image). A third image1512 is an example of an artificial fluorescent image generated using athree-model with three neural networks that can each receive thebrightfield image and output a one-channel fluorescent image (e.g., a256×256×1 image). A fourth image 1516 is an example of a greyscale errormap between the second image 1508 and the ground truth image 1504. Afifth image 1520 is an example of a greyscale error map between thethird image 1512 and the ground truth image 1504. A sixth image 1524, aseventh image 1528, an eighth image 1532, a ninth image 1536, and atenth image 1540 are examples of a zoomed-in organoid in the first image1504, the second image 1508, the third image 1512, the fourth image1516, and the fifth image 1520, respectively.

Next, the effect of adding SSIM loss to the 1-model objective functionwas elaborated. The objective function in Equation 1 is a weightedcombination of L1 and SSIM loss (λL1+βSSIM). The influence of SSIM wastested by uniformly evaluating β={0; 0.25; 0.5; 0.75; 1}. Table 2highlights the performance on the held-out test set using different β. Acombination of β=0.75 (and λ=0.25), shows the best performance of thetrained model (e.g., the trained model 508) in terms of both SSIM andRMSE.

TABLE 2 Experiment Avg. RMSE Avg. SSIM β = 0 1.39952 0.92383 β = 11.49570 0.91110 β = 0.25 1.37369 0.92691 β = 0.5 1.36477 0.92781 β =0.75 1.35165 0.92829

To determine if the accuracy of the trained model (β=0.75) was driven byspecific improvement in the prediction of a single fluorophore, such asprediction of DAPI (all cells) or FITC (dying/apoptotic cells), theaverage RMSE and SSIM across each channel were examined. The results areshown in Table 3.

TABLE 3 Channel Avg. RMSE Avg. SSIM Dead Cells 1.6311 0.92306 Dead/DyingCells 1.1015 0.92759 All Cells 1.7083 0.91807

The model trained with β=0.75 demonstrated consistent RMSE and SSIMscores across all channels. The performance of how the model trainedwith β=0.75 performed on two new patient colorectral cancer organoidlines (organoids lines B and C) was evaluated. Each new line had a totalof 648 brightfield and corresponding fluorescent readouts acrossdifferent plates. Table 4 demonstrates that a model trained on β=0.75trained on a single organoid can transfer to other organoid lines.However, the difference between the two lines suggests some limitations.The difference between the two lines suggests that different colorectalcancer organoid lines may present different morphological features thatmay limit model transfer. In that event, retraining the current bestmodel with some data from organoid line C or employing domain adaptationtechniques can facilitate better generalizability to organoid line C.

TABLE 4 Experiment Avg. RMSE Avg. SSIM Organoid line B 1.42222 0.91062Organoid line C 2.03384 0.78431

Example 2

Experiments were performed to try and improve model performance.Candidate models included a GANLoss+SSIM model, a GANLoss+SSIM+L1 modeltrained using a GANLoss+0.17L1+0.83 SSIM model, a GANLoss+MS-SSIM model,and a GANLoss+0.83MS-SSIM+0.17L1 model.

Initially, three separate Pix2Pix models were employed to train theindividual fluorescent channels. The Avg SSIM and RMSE results over thesame 743 blind test images as described in Example 1 are shown below.Tables 5-8 show results of the candidate models implemented inthree-model fashion. Table 5 shows results of the GANLoss+SSIM model.Table 6 shows results of the GANLoss+0.83SSIM+0.17L1 model. Table 7shows results of the GANLoss+MS-SSIM model. Table 8 shows results of theGANLoss+0.83 MS-SSIM+0.17 L1 model.

TABLE 5 Experiment Avg RMSE Avg SSIM Dead Cells 1.65133 0.91736Dead/Dying Cells 1.12827 0.91784 All cells 1.73630 0.91097

TABLE 6 Experiment Avg RMSE Avg SSIM Dead cells 1.56781 0.92950Dead/Dying Cells 1.10735 0.92782 All cells 1.70604 0.92024

TABLE 7 Experiment Avg RMSE Avg SSIM Dead cells 1.64130 0.92027Dead/Dying Cells 1.13677 0.91956 All cells 1.72046 0.91725

TABLE 8 Experiment Avg RMSE Avg SSIM Dead cells 1.61705 0.92530Dead/Dying Cells 1.12485 0.92856 All cells 1.72099 0.91723

Fluorescent Combined 3 Channel Image Results

The results in Table 9 below take the 3 channel pix2pix models perexperiment and combine them to form their 3 channel IF counterpart. Theindividual channels were trained separately and combined by stackingRGB.

TABLE 9 Experiment Avg RMSE Avg SSIM GANLoss + L1 1.36550 0.92299GANLoss + SSIM 1.38915 0.91558 GANLoss + SSIM + L1 1.33759 0.92586GANLoss + MS-SSIM 1.39169 0.91875 GANLoss + MS-SSIM + L1 1.37136 0.92372It was observed that GANLoss+SSIM or GANLoss+MS-SSIM standalone do notperform as well as other models. A combination of GANLoss+0.83SSIM+0.17L1 seems to perform the best. It was also found that GANLoss+L1and GANLoss+SSIM do not do a good job with detecting blurry bad qualityimages. The GANLoss+SSIM+L1 model was able to accurately detect blurryartifacts. The GANLoss+SSIM+L1 model recognized artifacts and blursbetter than other models and avoided prediction altogether whenblurs/artifacts are present in the brightfield image.

Example 3

In Example 2, the process of training 3 separate pix2pix models formultiple different objective functions proved to require several GPU's(3 per model) and extra effort in data curation. A similar performanceanalysis was done to check if similar/better RMSE and SSIM values wereobserved by directly training from brightfield to 3 channel fluorescenceusing a single Pix2Pix model in an attempt to reduce GPU usage.

Table 10 below shows the results of directly training to transfer styleto IF image for the same set of objective functions on the same test setof 743 images belonging to 10245. The number of GPU's was reduced from15 GPU's to 5 GPU's and the performance although not too significant, ismarginally better. Thus, it may be preferable to use a one-model togenerate artificial fluorescent images because performance can be atleast as good as a three-model, with one third of the computationalrequirements. In particular, a one-model trained using an objectivefunction of GANLoss+0.83 MS-SSIM+0.17 L1 model may outperform otherone-models and/or three-models trained on the same training data.

TABLE 10 Experiment Avg RMSE Avg SSIM GANLoss + L1 1.39952 0.92383GANLoss + SSIM 1.49570 0.91110 GANLoss + SSIM + L1 1.35567 0.92890GANLoss + MS-SSIM 1.44965 0.91841 GANLoss + MS-SSIM + L1 1.39880 0.92577

Table 11 below shows results of a number of one-models trained usingdifferent objective functions. GANLoss+0.75 SSIM+0.25 L1 had the bestRMSE, while GANLoss+0.83 SSIM+17 L1 had the best SSIM performance.

TABLE 11 Experiment Avg RMSE Avg SSIM GANLoss + 0.5 SSIM + 0.5 L11.36478 0.92781 GANLoss + 0.5 MSSSIM + 0.5 L1 1.40834 0.92434 GANLoss +0.17 SSIM + 0.83 L1 1.34783 0.92641 GANLoss + 0.17 MSSSIM + 0.83 L11.37889 0.92560 GANLoss + 0.25 SSIM + 0.75 L1 1.37369 0.92691 GANLoss +0.25 MSSSIM + 0.75 L1 1.37788 0.92547 GANLoss + 0.75 SSIM + 0.25 L11.35166 0.92830 GANLoss + 0.75 MSSSIM + 0.25 L1 1.39788 0.92541GANLoss + 0.83 SSIM + 0.17 L1 1.35567 0.92890 GANLoss + 0.83 MSSSIM +0.17 L1 1.39880 0.92577

Example 4

This example details an exemplary cell profiler readout. The cellprofiler readout includes all cell counts and dead cell counts of realfluorescent images and corresponding artificial fluorescent images. InTable 12, each row indicates a particular site in a well within anexperimental plate and whether it is an artificial or a real imageimaged from an ImageXpress Micro microscope.

TABLE 12 Count_ Count_PrimaryDead Cells Cells FileName_NativeImageNumber 281.0 170.0 Assay10B_10245- 1 10301_A03_s1_ fake B.png 295.0107.0 Assay10B_10245- 2 10301_A03_s1_ real_B.png 269.0 211.0Assay10B_10245- 3 10301_A20_s2_ fake_B.png 270.0 210.0 Assay10B_10245- 410301_A20_s2_ real_B.png 549.0 162.0 Assay10B_10245- 5 10301_C20_s1_fake_B.png

Table 13 below shows plate and well information for each image alongwith the SSIM/RMSE values.

TABLE 13 Count_Primary file Count_Cells DeadCells file Name type platewell welltype SSIM RMSE 281.0 170.0 [Assay10B, 10245- Assay10B_10245-fake Assay10B A03 A 0.947745 1.165460 10301, A03, s1, fake, B]10301_A03_s1 295.0 107.0 [Assay10B,10245- Assay10B_10245- real Assay10BA03 A 0.947745 1.165460 10301, A03, s1, real, B] 10301_A03_s1 269.0211.0 [Assay10B, 10245- Assay10B_10245- fake Assay10B A20 A 0.9568551.011834 10301, A20, s2, fake, B] 10301_A20_s2 270.0 210.0[Assay10B,10245- Assay10B_10245 real Assay10B A20 A 0.956855 1.01183410301, A20, s2, real, B] 10301_A20_s2 549.0 162.0 [Assay10B, 10245-Assay10B_10245- fake Assay10B C20 C 0.940647 1.194703 10301, C20, s1,fake, B] 10301_C20_s1

Table 13 shows that the fluorescent images can produce similar cellcounts as compared to the corresponding real fluorescent image.

Example 5

In some embodiments, a large scale drug assay in tumor organoids canincrease throughput of the assay. This high throughput screening can beused for validation or testing of drug efficacy or for discovery ofnovel therapeutics. In some embodiments, 3D TOs may be more similar to atumor from which they are derived than a 2-dimensional clonalestablished cell line derived from that tumor.

In this example, tumor tissue removed by a biopsy is dissociated intosingle cells and grown into a 3-dimensional (3D) tumor organoid (TO)culture including TOs. TOs are then dissociated into single cells andgrown in a 384-well tissue culture plate for 72 hours. Each wellreceives either no treatment (or a mock treatment) or a dose(concentration) of a small molecule inhibitor or chemotherapy drugs andthe effect of the treatment on the cells in the TO is measured. In oneexample, over 1,000 drugs may be tested. In another example, severalconcentrations of 140 drugs may be tested.

In one example, the treatment is one of three hundred and fifty-onesmall molecule inhibitors and seven doses are tested for each treatmenton two different organoid types (two organoid cell lines), each derivedfrom a separate patient sample. In this example, one organoid type is agastric cancer organoid line and the other is a colorectal cancerorganoid line. In one example, the effect of the treatment may bemeasured by counting the number of dead cells and/or viable cells in awell after exposure to a treatment. In this example of fluorescentstaining, cell nuclei are stained blue with Hoechst 33342, dying(apoptotic) cells are stained green with Caspase-3/7 Apoptosis AssayReagent, and dead cells are stained red with TO-PRO-3.

In this example, the gastric cancer organoid line has an amplificationof the HER2 gene. Afatinib (a drug that targets HER2, among othermolecules) and two other drugs that target HER2 kill this gastric cancerorganoid line effectively.

In some embodiments, the methods and systems described above may beutilized in combination with or as part of a digital and laboratoryhealth care platform that is generally targeted to medical care andresearch. It should be understood that many uses of the methods andsystems described above, in combination with such a platform, arepossible. One example of such a platform is described in U.S. patentapplication Ser. No. 16/657,804, titled “Data Based Cancer Research andTreatment Systems and Methods”, and filed Oct. 18, 2019, which isincorporated herein by reference and in its entirety for all purposes.

For example, in some embodiments of the methods and systems describedabove may include microservices constituting a digital and laboratoryhealth care platform supporting artificial fluorescent image generationand analysis. Some embodiments may include a single microservice forexecuting and delivering artificial fluorescent image generation or mayinclude a plurality of microservices each having a particular role whichtogether implement one or more of the embodiments above. In one example,a first microservice may execute training data generation in order todeliver training data to a second microservice for training a model.Similarly, the second microservice may execute model training to delivera trained model according to at least some embodiments. A thirdmicroservice may receive a trained model from a second microservice andmay execute artificial fluorescent image generation.

Some embodiments above can be executed in one or more microservices withor as part of a digital and laboratory health care platform, one or moreof such micro-services may be part of an order management system thatorchestrates the sequence of events as needed at the appropriate timeand in the appropriate order necessary to instantiate embodiments above.A micro-services based order management system is disclosed, forexample, in U.S. Prov. Patent Application No. 62/873,693, titled“Adaptive Order Fulfillment and Tracking Methods and Systems”, filedJul. 12, 2019, which is incorporated herein by reference and in itsentirety for all purposes.

For example, continuing with the above first and second microservices,an order management system may notify the first microservice that anorder for training a model has been received and is ready forprocessing. The first microservice may execute and notify the ordermanagement system once the delivery of training data is ready for thesecond microservice. Furthermore, the order management system mayidentify that execution parameters (prerequisites) for the secondmicroservice are satisfied, including that the first microservice hascompleted, and notify the second microservice that it may continueprocessing the order to generate a trained model according to someembodiments.

When the digital and laboratory health care platform further includes areport generation engine, the methods and systems described above may beutilized to create a summary report of a patient's genetic profile andthe results of one or more insight engines for presentation to aphysician. For instance, the report may provide to the physicianinformation about the extent to which a specimen that was used toharvest organoids. For example, the report may provide a genetic profilefor each of the tissue types, tumors, or organs in the specimen. Thegenetic profile may represent genetic sequences present in the tissuetype, tumor, or organ and may include variants, expression levels,information about gene products, or other information that could bederived from genetic analysis of a tissue, tumor, or organ. The reportmay include therapies and/or clinical trials matched based on a portionor all of the genetic profile or insight engine findings and summaries.For example, the therapies may be matched according to the systems andmethods disclosed in U.S. Prov. Patent Application No. 62/804,724,titled “Therapeutic Suggestion Improvements Gained Through GenomicBiomarker Matching Plus Clinical History”, filed Feb. 12, 2019, which isincorporated herein by reference and in its entirety for all purposes.For example, the clinical trials may be matched according to the systemsand methods disclosed in U.S. Prov. Patent Application No. 62/855,913,titled “Systems and Methods of Clinical Trial Evaluation”, filed May 31,2019, which is incorporated herein by reference and in its entirety forall purposes.

The report may include a comparison of the results to a database ofresults from many specimens. An example of methods and systems forcomparing results to a database of results are disclosed in U.S. Prov.Patent Application No. 62/786,739, titled “A Method and Process forPredicting and Analyzing Patient Cohort Response, Progression andSurvival”, and filed Dec. 31, 2018, which is incorporated herein byreference and in its entirety for all purposes. The information may beused, sometimes in conjunction with similar information from additionalspecimens and/or clinical response information, to discover biomarkersor design a clinical trial.

When the digital and laboratory health care platform further includesapplication of one or more of the embodiments herein to organoidsdeveloped in connection with the platform, the methods and systems maybe used to further evaluate genetic sequencing data derived from anorganoid to provide information about the extent to which the organoidthat was sequenced contained a first cell type, a second cell type, athird cell type, and so forth. For example, the report may provide agenetic profile for each of the cell types in the specimen. The geneticprofile may represent genetic sequences present in a given cell type andmay include variants, expression levels, information about geneproducts, or other information that could be derived from geneticanalysis of a cell. The report may include therapies matched based on aportion or all of the deconvoluted information. These therapies may betested on the organoid, derivatives of that organoid, and/or similarorganoids to determine an organoid's sensitivity to those therapies. Forexample, organoids may be cultured and tested according to the systemsand methods disclosed in U.S. patent application Ser. No. 16/693,117,titled “Tumor Organoid Culture Compositions, Systems, and Methods”,filed Nov. 22, 2019; and U.S. Prov. Patent Application No. 62/924,621,which are incorporated herein by reference and in their entirety for allpurposes.

When the digital and laboratory health care platform further includesapplication of one or more of the above in combination with or as partof a medical device or a laboratory developed test that is generallytargeted to medical care and research, such laboratory developed test ormedical device results may be enhanced and personalized through the useof artificial intelligence. An example of laboratory developed tests,especially those that may be enhanced by artificial intelligence, isdisclosed, for example, in U.S. Provisional Patent Application No.62/924,515, titled “Artificial Intelligence Assisted Precision MedicineEnhancements to Standardized Laboratory Diagnostic Testing”, and filedOct. 22, 2019, which is incorporated herein by reference and in itsentirety for all purposes.

It should be understood that the examples given above are illustrativeand do not limit the uses of the systems and methods described herein incombination with a digital and laboratory health care platform.

The systems and methods disclosed herein can reduce (1) the timerequired for imaging the plates (2) the need for toxic dyes that couldcause cell death and skew results and (3) the amount of manual labor toadd those dyes, allowing larger numbers of drugs or concentrations ofdrugs to be tested (for example, by a factor of ten). The systems andmethods reduce the number of images generated to analyze each plate (forexample, by a factor of four or 5, or from 10,000 images to about2,000-2,500 images) by receiving a brightfield image and predicting thecorresponding fluorescent readout and allowing label-free (dye-free)estimation of cell viability (for example, the percentage of cells in awell or in an image that are alive at a given time) at multiple timepoints.

The systems and methods described herein may be used to make hundreds ormore measurements in each cell culture well to assess heterogeneity ofthe drug response (for example, surviving or dying after treatment) ofthe organoids, which may be done on a per organoid basis (for example,analyzing the cell death or percent of viable cells in each organoid).Multiple measurements include fluorescence intensity, cell growth, celldeath, cells per organoid, cells per well, dose response (for example,graphing % cell viability vs. drug dose, calculating a best fit curve ordrug dose response curve, and measuring the area above the curve andbelow the 100% viability intercept), etc. These measurements facilitatedetermination of cellular mechanisms of a drug response and/or thedetection of drug-resistant subpopulations of organoids within anorganoid line or within a cell culture well. Drug efficacy (for example,dose response) and specificity may be measured. Drug efficacy of alldrugs for one or more organoid lines may be plotted. In one example, thex-axis shows drug efficacy for a first organoid line and the y-axisshows drug efficacy for a second organoid line. In this plot, drugs nearthe upper right corner were effective against both organoid lines. Drugsnear the upper left or lower right corners were effective against one ofthe organoid lines.

Drugs that kill an organoid line may also be categorized and quantifiedaccording to their drug target. For example, the thirty most effectivedrugs that kill an organoid line may be organized by target in a bargraph, showing the number of effective drugs per target.

Referring now to FIG. 16 , a flow for generating an artificialfluorescent image 1616 using a first trained model 1604 and a secondtrained model 1612 is shown. The first trained model 1604 can generateone or more individual organoids 1608 (e.g., organoid segmentations)based on a brightfield image 1600. The brightfield image 1600 maycontain one or more organoids, and the trained first model 1604 canidentify each organoid by segmenting the organoids 1608 from thebrightfield image 1600. In some embodiments, the first trained model caninclude an artificial neural network. In some embodiments, theartificial neural network can include a Mask-RCNN network.

In some embodiments, the first trained model 1604 and the second trainedmodel can be used to predict drug response and other characteristics ofan organoid line based on viable cells and/or morphology associated witheach individual tumor organoid (TO) in the brightfield image 1600.

Assessing each organoid individually may provide better informationabout treatments than if the organoids are assessed in place in thebrightfield image 1600. Each TO may represent a different tumor clonepresent in the specimen. Each TO may exhibit a different therapeuticresponse to the different drugs at different dosage levels. Instead ofassessing viabilities of the TOs by aggregating the viabilities acrossthe entire field-of-view in an image, understanding the distribution ofthe viabilities at a per-organoid level (for example, how many cells ineach organoid are viable) and possibly aggregating the viabilities ofthe TOs belonging to the same tumor clone may offer a betterunderstanding of the response of organoids to the drugs, and byextension, a better understanding of the response of a patient to thedrugs.

In some embodiments, the first trained model 1604 can be trained tosegment organoids out from the brightfield image 1600 using a trainingset of brightfield images annotated with bounding boxes around theindividual organoids. In some embodiments, the first trained model 1604can generate masks and bounding boxes around every organoid in thebrightfield image. In some embodiments, the first trained model 1604 cangenerate model embeddings that can be used to generate features based onthe organoids in order to assess viability and morphology.

In some embodiments, the second trained model 1612 can include thegenerator 408 trained to generate an artificial fluorescent image basedon an input brightfield organoid. The second trained model 1612 can betrained on a training set of individual brightfield organoids andindividual fluorescent organoids. Each individual fluorescent organoidcan be used to generate a viability. The viabilities for all organoidscan be aggregated. A distribution of viabilities per organoid can begenerated and/or visualized. In some embodiments, the distribution oflive/dead cells per organoid can be calculated to get a prediction orextrapolation of dose response from the distribution of Organoidviability at a single drug concentration. In some embodiments, a processcan aggregate the viabilities of different tumor clones among theorganoids if side information is available to determine which croppedout TO belongs to which tumor clone.

In some embodiments, the morphologies of every organoid can bevisualized. In some embodiments, the morphologies of the tumor organoidscan be visualized, either by using handcrafted features or modelembeddings, and clustering in a supervised or unsupervised setting. Insome embodiments, the morphological clusters can be associated withcluster viabilities, and by extension, drug response. In someembodiments, the TO morphology can be used to predict drug response.

Referring now to FIG. 16 as well as FIG. 17 , a process 1700 forgenerating fluorescent images of tumor organoids is shown. The process1700 can be implemented as computer readable instructions on one or morememories or other non-transitory computer readable media, and executedby one or more processors in communication with the one or more memoriesor other media. In some embodiments, the process 1700 can be implementedas computer readable instructions on the memory 220 and/or the memory240 and executed by the processor 204 and/or the processor 224. In someembodiments, the process 1700 can be executed by an imaging system. Insome embodiments, a brightfield microscopy imaging system can beconfigured to execute the process 1700. In some embodiments, thebrightfield microscopy imaging system can include one or more memoriesor other non-transitory computer readable media including the process1700 implemented as computer readable instructions on the one or morememories or other non-transitory computer readable media, and one ormore processors in communication with the one or more memories or othermedia configured to execute the computer readable instructions toexecute the process 1700.

At 1704, the process 1700 can receive a brightfield image (e.g., thebrightfield image 1600 in FIG. 16 ) of one or more organoids. In someembodiments, the brightfield image can be preprocessed in order toenhance contrast as described above. In some embodiments, thebrightfield image can be a raw image that has not undergone anypreprocessing such as contrast enhancement.

At 1708, the process 1700 can determine if the brightfield image isunprocessed (i.e., raw). If the brightfield image is unprocessed (i.e.,“YES” at 1708), the process 1700 can proceed to 1712. If the brightfieldimage is not unprocessed (i.e., “NO” at 1708), the process 1700 canproceed to 1716.

At 1712, the process 1700 can preprocess the brightfield image. In someembodiments, the brightfield image can have pixel intensities rangingfrom [0, 2¹⁶]. In some embodiments, the process 1700 can convert thebrightfield image to an unsigned byte format, with values ranging from[0, 255]. In some embodiments, the process 1700 can convert thebrightfield image to another format with less bits than the originalpixel intensity. The process 1700 can then stretch and clip each pixelintensity to a desired output range. In some embodiments, the process1700 can determine an input range for the brightfield image by uniformlystretching the 2nd and 98th percentile of pixel intensities in thebrightfield image to an output range [0,255].

At 1716, the process 1700 can provide the brightfield image to a firsttrained model. In some embodiments, the first trained model can be thefirst trained model 1604 in FIG. 16 . In some embodiments, the trainedmodel can a neural network. In some embodiments, the neural network caninclude a Mask-RCNN model.

At 1720, the process 1700 can receive at least one individual tumororganoid from the first trained model (for example, in a 64×64×1 or32×32×1 image). Each individual tumor organoid can be a portion of thebrightfield image.

At 1724, the process 1700 can provide the at least one individual tumororganoid to a second trained model. In some embodiments, the secondtrained model can include the second trained model 1612 in FIG. 16 . Insome embodiments, the process 1700 can sequentially provide eachindividual tumor organoid to the second trained model. In someembodiments, the process 1700 can apply the second trained model to theat least one individual tumor organoid to generate at least oneartificial fluorescent image.

At 1728, the process 1700 can receive at least one artificialfluorescent image from the second trained model. Each artificialfluorescent image can be generated based on an individual tumororganoid. The artificial fluorescent image can indicate whether cellsincluded in the tumor organoids are alive or dead.

At 1732, the process 1700 can cause the at least one artificialfluorescent image to be output. In some embodiments, the process 1700can cause the at least one artificial fluorescent image to be output toat least one of a memory (e.g., the memory 220 and/or the memory 240)and/or a display (e.g., the display 116, the display 208, and/or thedisplay 228). The at least one artificial fluorescent image can be usedto provide a live/dead count of cells in each individual organoid. Insome embodiments, the process 1700 can cause the at least one artificialfluorescent image to be output to an automatic cell counting process inorder to receive an accurate live/dead count of cells, percentage ofcells that are viable, and/or a cell count report for each organoid. Forexample, the process 1700 can cause the at least one artificialfluorescent image to be output to the CellProfiler available athttps://cellprofiler.org. In some embodiments, the process 1700 cancause one or more channels of the at least one artificial fluorescentimage to be output to an automatic cell counting process in order toreceive a cell count report, percentage of cells that are viable, and/oraccurate live/dead count of cells in each organoid. In some embodiments,the process 1700 can cause the artificial fluorescent image to be outputto a trained model in order to receive a cell count report, percentageof cells that are viable, and/or accurate live/dead count of cells inthe artificial fluorescent image. In some embodiments, the process 1700can cause a combination (e.g., image embeddings combined byconcatenation) of the brightfield image and one, two, or three channelsof the artificial fluorescent image to be output to an automatic cellcounting process in order to receive a cell count report, percentage ofcells that are viable, and/or an accurate live/dead count of cells inthe artificial fluorescent image.

In some embodiments, at 1732, the process 1700 can identify cells in theartificial fluorescent image by converting each of the channels tograyscale, enhancing and suppressing certain features such as speckles,ring shapes, neurites, dark holes, identifying primary objects belongingto the all cell channel where the typical diameters of these objects (inpixel units) is set anywhere between 2 and 20 with a minimum crossentropy thresholding method at a smoothing scale of 1.3488, andidentifying primary objects again belonging to the dead cells channelwhere typical diameter is anywhere between five and twenty in pixelunits. In this way, the process 1700 can generate a cell count report.In some embodiments, the process 1732 can determine if a drug and/ordosage is effective in killing tumor organoid cells based on thelive/dead count of cells or percentage of cells that are viable for eachorganoid. In some embodiments, at 1732, the process 1700 can extrapolatedose response from a distribution of organoid viability at a singleconcentration.

In some embodiments, the process 1700 can analyze nucleic acid dataassociated with the one or more tumor organoids. Each tumor organoidincluded in the one or more tumor organoids can be associated with aspecimen (e.g., the specimen the tumor organoid was harvested from). Insome embodiments, each specimen can be associated with a patient. Thepatient can be associated with patient data that can include nucleicacid data. In some embodiments, the nucleic acid data can include wholeexome data, transcriptome data, DNA data, and/or RNA data. The nucleicacid data may be used to further analyze the patient. In someembodiments, the process 1700 can associate the artificial fluorescentimage with information about the specimen (e.g., the nucleic acid data).In some embodiments, the process 1700 can provide the artificialfluorescent image and the associated information about the specimen to adatabase. In some embodiments, the database can include at least sevenhundred and fifty artificial fluorescent images.

In some embodiments, the process 1700 can generate a report based on thecell count, the cell count report, the nucleic acid data and/or theartificial fluorescent image. In some embodiments, the process 1700 cancause the report to be output to at least one of a memory (e.g., thememory 220 and/or the memory 240) and/or a display (e.g., the display116, the display 208, and/or the display 228). The process 1700 can thenend.

Example 6—Neural Network-Based Model for Predicting TO Drug Response,and Response Prediction from Brightfield Images in the Absence ofFluorescent Labels

In some embodiments, a process can cause a brightfield image and one,two, or three channels of the artificial fluorescent image to be outputto an automatic cell counting process (for example, a viabilityestimation process) in order to receive a percentage of cells in theimage that are viable (alive).

FIG. 18 illustrates a flow 1800 for predicting a viability 1820 based ona brightfield image 1804. The brightfield image 1804 can be athree-channel brightfield image of tumor organoids and/or and cells. Theflow 1800 can include providing the brightfield image 1804 to agenerator 1808. In some embodiments, the generator 1808 can generate anartificial fluorescent image 1812 based on the brightfield image 1804.The flow 1800 can include providing the brightfield image 1804 and theartificial fluorescent image 1812 to a discriminator 1816. Thediscriminator can generate the viability 1820 based on the brightfieldimage 1804 and the artificial fluorescent image 1812.

Referring now to FIG. 18 as well as FIG. 19 , an exemplary generator1900 and an exemplary discriminator 1902 are shown. In some embodiments,the discriminator 1902 can be used to train the generator 1900. In someembodiments, the generator 1900 and the discriminator 1902 can beincluded in a regularized conditional adversarial (RCA) network.

In some embodiments, the generator 1900 can include an encoder-decoderU-Net network. In some embodiments, the U-Net can include skipconnections. In some embodiments, the generator 1900 can receive atwo-dimensional brightfield image (e.g., a 1024×1024 brightfield image).In some embodiments, the generator 1900 can generate a normalized,three-channel, high-resolution 1024×1024×3 output fluorescent imagebased on the brightfield image, where the three channels correspond toHoechst 33342 all nuclei stained readout, Caspase-3/7 apoptotic stainedreadout, and TOPRO-3 dead cell stained readout, respectively.

Referring now to FIGS. 18 and 19 as well as FIG. 20 , a discriminator1904 can generate a viability prediction 1924 based on a brightfieldimage and an artificial fluorescent image. The discriminator 1904 caninclude an encoder branch 1908 and a fully-connected branch 1912. Insome embodiments, the encoder branch 1908 can include a 70×70 patchGAN.In some embodiments, the encoder branch 1908 can receive a concatenatedbrightfield image and a fluorescent image 1916 of size 1024×1024×6. Insome embodiments, the encoder branch 1908 can generate an outputprediction map 1920 (e.g., an output prediction map of size 126×126×1).The fully-connected branch 1912 can then generate a viability prediction1924 based on the output prediction map 1920. In some embodiments, thefully-connected branch 1912 can include a number of fully-connectedlayers (e.g., two fully-connected layers) and a sigmoid activation layerthat outputs the viability prediction 1924. The viability prediction1924 can indicate viability. In some embodiments, the viabilityprediction 1924 can be and/or range from zero (indicative of noviability) to one (indicative of high viability).

Training

In testing, the generator 1900 and the discriminator 1902 were trainedon eight thousand four hundred and fifteen paired brightfield and3-channel fluorescence images from colon adenocarcinoma TO screeningexperiments, each with associated calculated drug responses based onTO-PRO-3 viability. In some embodiments, an objective function (forexample, loss function used for training) can include an additional meansquared error loss in a discriminator objective to regress against thebranch of the discriminator that computes overall viability perbrightfield image. Exemplary loss functions for the discriminator 1902and the generator 1900 are given below:

D _(Loss)=MSE_(Loss){Real Prediction,1}+MSE_(Loss){FakePrediction,0}+MSE_(Loss){Predicted Viability,Viability}

G _(Loss)=MSE_(Loss){Fake Prediction,1}+MAE_(Loss){Fake Fluorescent,RealFluorescent}+SSIM{Fake Fluorescent,Real Fluorescent}

Weights for the discriminator 1902 can be updated by minimizingD_(Loss), and weights for the generator 1900 can be updated bymaximizing G_(LOSS).

Validation

In validation, representative images of real versus generatedfluorescence demonstrated nearly indistinguishable visual matching.These results were confirmed using two quantitative metrics: thestructural similarity index (SSIM) as well as the root mean squarederror (RMSE). The reported average SSIM and RMSE values across 1,526samples of the colon adenocarcinoma TO used in the screening experimentwere 0.90 and 0.13924 respectively. For the gastric TO line, thereported average SSIM and RMSE values across 9200 samples were 0.898 and0.136, respectively. TO description, image analysis, and generation ofimages for training data

TOs were dissociated into single cells and resuspended in a 30:70% mixof GFR Matrigel:growth media at a concentration of 100 cells/μl. Thesolution was added to 384-well assay plates (Corning) at 20 μl per wellfor a final concentration of 2,000 cells per well. Assay plates werecovered with a Breathe-Easy sealing membrane (Sigma Aldrich) to preventevaporation. TOs were grown for 72 hours before drug addition. Drugswere prepared in growth media with 2.5 μM Caspase-3/7 Green ApoptosisAssay Reagent (Essen Bioscience). Serial dilutions of each molecule wereprepared in 384-well polystyrene plates (Nunc). Seven 10-fold dilutionswere made for each compound with the high dose being 10 μM. Selectcompounds were limited to a high dose of 1 μM by maximum solubility.Diluted drug was added to the assay plate using an Integra Viaflopipette (Integra) mounted on an Integra Assist Plus Pipetting Robot(Integra). Assay plates were again covered with a Breathe-Easy sealingmembrane and TOs were exposed to drugs for another 72 hours beforeimaging.

Prior to imaging, TOs were incubated with 4 μM Hoechst 33342 (FisherScientific) and 300 nM TO-PRO-3 Iodide (642/661) (Invitrogen) for 1.5-2hours. Assay plates were imaged using an ImageXpress Micro Confocal(Molecular Devices) at 10× magnification so that ˜100-200 TOs wereimaged per well. The multiplexed fluorescence images were 1024×1024×3RGB images, where red corresponds to dead cells (TO-PRO-3), green toapoptotic cells (Caspase-3/7), and blue to nuclei (Hoechst 33342). Allwavelength channels underwent a simple intensity rescaling contrastenhancement technique to brighten and sharpen the TOs/cells as well asremove background noise.

Images were acquired as 4×15 μm Z-stacks and the 2D projections wereanalyzed to assess cell viability. Confocal images were analyzed usingthe MetaXpress software (Molecular Devices) custom module editor featureto design an analysis module that identified TOs by clusters of Hoechst33342 staining, individual cells by Hoechst 33342 staining, anddead/dying cells by either TO-PRO-3 or Caspase-3/7 staining. The resultof this analysis module is a spreadsheet detailing the number of liveand dead cells for every individual organoid. Each viability value wasequal to or greater than 0 (0% of cells viable) and equal to or lessthan 1 (100% of cells viable).

Viability calculation=sum total of all live cells in the site/sum totalof all cells in the site (gives a proportion of live cells per site).More effective drugs will have lower viabilities (more cells die) athigher doses compared to less effective drugs, which have higherviabilities.

The mean viability for all organoids per site (for example, per image)was obtained from the MetaXpress software readout. For each image addedto a training data set used to train the viability discriminator, theimage was stored with the mean viability associated with that image as alabel or metadata. The images had a resolution of 1024×1024 and wererandomly flipped as a data augmentation step before being used astraining data.

The training data set in this example included seven thousand imagesrepresenting 15 culture plates.

The percentage of viable cells per TO was calculated based on the imageanalysis described above. TOs with fewer than three cells, TOs largerthan the top one percent by size, and wells with fewer than 20 TOsdetected were excluded from analysis.

In another example, an AUC may be used as metadata or a label togenerate training data. The mean viability for all TOs at a given drugconcentration was used in dose-response curves to calculate AUC. AUC wascalculated using the computeAUC function using settings for “actual” AUCof the R Package PharmacoGx (v1.17.1). Heatmaps of AUC values weregenerated using the Pheatmap package (v1.0.12) in R. Scatterplots of AUCvalues were generated using the ggplot2 package (v3.3.0) in R.

Referring now to FIGS. 18-20 as well as FIG. 21 , a process 2100 forgenerating a viability value is shown. The process 2100 can beimplemented as computer readable instructions on one or more memories orother non-transitory computer readable media, and executed by one ormore processors in communication with the one or more memories or othermedia. In some embodiments, the process 2100 can be implemented ascomputer readable instructions on the memory 220 and/or the memory 240and executed by the processor 204 and/or the processor 224. In someembodiments, the process 2100 can be executed by an imaging system. Insome embodiments, a brightfield microscopy imaging system can beconfigured to execute the process 2100. In some embodiments, thebrightfield microscopy imaging system can include one or more memoriesor other non-transitory computer readable media including the process2100 implemented as computer readable instructions on the one or morememories or other non-transitory computer readable media, and one ormore processors in communication with the one or more memories or othermedia configured to execute the computer readable instructions toexecute the process 2100.

At 2104, the process 2100 can receive a brightfield image (e.g., thebrightfield image 1804 in FIG. 18 ) of one or more organoids. In someembodiments, the brightfield image can be preprocessed in order toenhance contrast as described above. In some embodiments, thebrightfield image can be a raw image that has not undergone anypreprocessing such as contrast enhancement.

At 2108, the process 2100 can determine if the brightfield image isunprocessed (i.e., raw). If the brightfield image is unprocessed (i.e.,“YES” at 2108), the process 2100 can proceed to 2112. If the brightfieldimage is not unprocessed (i.e., “NO” at 2108), the process 2100 canproceed to 2116.

At 2112, the process 2100 can preprocess the brightfield image. In someembodiments, the brightfield image can have pixel intensities rangingfrom [0,2{circumflex over ( )}16]. In some embodiments, the process 2100can convert the brightfield image to an unsigned byte format, withvalues ranging from [0, 255]. In some embodiments, the process 2100 canconvert the brightfield image to another format with less bits than theoriginal pixel intensity. The process 2100 can then stretch and clipeach pixel intensity to a desired output range. In some embodiments, theprocess 2100 can determine an input range for the brightfield image byuniformly stretching the 2nd and 98th percentile of pixel intensities inthe brightfield image to an output range [0,255].

At 2116, the process 2100 can provide the brightfield image to a trainedmodel. In some embodiments, the trained model can include a generator(e.g., the generator 1808 and/or the generator 1900) and thediscriminator (e.g., the discriminator 1816 and/or the discriminator1904). In some embodiments, the process 2100 can include providing thebrightfield image to the generator, receiving an artificial fluorescentimage from the process, concatenating the brightfield image with theartificial fluorescent image to generate a concatenated image, andproviding the concatenated image to the discriminator. In someembodiments, the process 2100 can include applying the trained model tothe brightfield image to generate the fluorescent image.

At 2120, the process 2100 can receive a viability (e.g., a viabilityvalue) from the trained model. In some embodiments, the process 2100 canreceive the viability from the discriminator 1904. In some embodiments,the viability can be the viability 1820 and/or the viability prediction1924.

At 2124, the process 2100 can cause the viability to be output. In someembodiments, the process 2100 can cause viability to be output to atleast one of a memory (e.g., the memory 220 and/or the memory 240)and/or a display (e.g., the display 116, the display 208, and/or thedisplay 228). In some embodiments, the process 2100 can generate areport based on the viability. In some embodiments, the process 2100 cananalyze nucleic acid data associated with the one or more organoids.Each organoid included in the one or more organoids can be associatedwith a specimen (e.g., the specimen the organoid was harvested from). Insome embodiments, each specimen can be associated with a patient. Thepatient can be associated with patient data that can include nucleicacid data. In some embodiments, the nucleic acid data can include wholeexome data, transcriptome data, DNA data, and/or RNA data. The nucleicacid data may be used to further analyze the patient. In someembodiments, the process 2100 can associate the artificial fluorescentimage with information about the specimen (e.g., the nucleic acid data).In some embodiments, the process 2100 can provide the artificialfluorescent image, the associated information about the specimen, and/orthe viability to a database. In some embodiments, the database caninclude at least seven hundred and fifty artificial fluorescent images.

In some embodiments, the process 2100 can cause the report to be outputto at least one of a memory (e.g., the memory 220 and/or the memory 240)and/or a display (e.g., the display 116, the display 208, and/or thedisplay 228). The process 2100 can then end.

The present disclosure has described one or more preferred embodiments,and it should be appreciated that many equivalents, alternatives,variations, and modifications, aside from those expressly stated, arepossible and within the scope of the invention.

1-30. (canceled)
 31. A method of generating an artificialimmunohistochemistry (IHC) stained image of cells, without IHC staining,comprising: receiving, by a computer system having at least oneprocessor, a hematoxylin and eosin (H&E) stained image generated by abrightfield microscopy imaging modality of at least a portion of cellsincluded in a specimen (brightfield H&E image) derived from a patient;applying by the processor, to the brightfield H&E image, at least onetrained model, wherein the trained model is trained to modify pixelintensities of the brightfield H&E image so as to generate theartificial IHC image of the cells based on the brightfield image; andgenerating the artificial IHC image by the trained model.
 32. The methodof claim 31, wherein the specimen comprises a tissue slice or a bloodsmear from the patient.
 33. The method of claim 32, wherein theartificial IHC image is indicative of whether the portion of the cellsincluded in the specimen are positive or negative for an IHC targetmolecule.
 34. The method of claim 32, wherein the specimen is associatedwith a colorectal cancer, a gastric cancer, a breast cancer, a lungcancer, an endometrial cancer, a colon cancer, a head and neck cancer,an ovarian cancer, a pancreatic cancer, a gastric cancer, ahepatobiliary cancer, or a genitourinary cancer.
 35. The method of claim32, further comprising: outputting the artificial IHC image to aprediction model to predict response of the patient to a treatmentmethod.
 36. The method of claim 32, wherein the specimen is included ina first group of specimens, and the method further comprises: providinga second brightfield H&E image to the trained model; and receiving asecond artificial IHC image from the trained model, wherein the secondbrightfield H&E image comprises a second group of tissue slices or bloodsmears, and the second artificial IHC image is indicative of whethercells included in the second group are positive or negative for the IHCtarget molecule.
 37. The method of claim 31, further comprising:providing to the trained model a plurality of brightfield H&E imagesgenerated by a brightfield microscopy imaging modality of the at least aportion of cells, the plurality of brightfield H&E images generatedafter the brightfield H&E image is generated; and receiving acorresponding plurality of artificial IHC images from the trained model.38. The method of claim 31 further comprising: generating a report basedon the artificial IHC image.
 39. The method of claim 31, wherein thetrained model is trained based on a loss function comprising adiscriminator loss, or a generator loss and a discriminator loss. 40.The method of claim 31 further comprising preprocessing the brightfieldH&E image to increase contrast levels.
 41. The method of claim 31further comprising preprocessing brightfield H&E images and IHC imagesincluded in training data used to train the trained model.
 42. Themethod of claim 31, wherein the trained model comprises a generator, thegenerator being trained in part by a discriminator.
 43. The method ofclaim 31, further comprising analyzing nucleic acid data associated withthe specimen, and including the results of the analysis in a report. 44.The method of claim 31, further comprising: associating the artificialIHC image with information about the specimen; and providing theartificial IHC image and the associated information about the specimento a database comprising at least seven hundred and fifty artificial IHCimages.
 45. The method of claim 44, wherein the information about thespecimen comprises a cancer diagnosis associated with the specimen. 46.The method of claim 44 further comprising: applying one or more drugs tothe patient prior to the generation of the brightfield H&E image;wherein the information about the specimen comprises a diagnosisassociated with the specimen and an identification of the one or moredrugs.
 47. A brightfield microscopy imaging modality configured toexecute the method of claim
 31. 48. The brightfield microscopy imagingmodality of claim 47, wherein the brightfield microscope imagingmodality comprises a brightfield microscope.
 49. A pathology slideanalysis system comprising at least one processor and at least onememory, the system configured to receive an H&E stained image generatedby a brightfield microscopy imaging modality from at least a portion ofcells included in a specimen (brightfield H&E image); apply, via theprocessor, to the brightfield H&E image, at least one model trained tomodify pixel intensities of the brightfield H&E image so as to generatean artificial IHC image based on the brightfield H&E image without IHCstaining, the artificial IHC image being indicative of whether the cellsincluded in the specimen are positive or negative for an IHC targetmolecule; generating the artificial IHC image by the trained model; andoutput the artificial IHC image to at least one of a memory or adisplay.
 50. A method of generating an artificial IHC image of cellswithout IHC stain, comprising; receiving, from a computer system havingat least one processor, an H&E stained image generated by a brightfieldmicroscopy imaging modality from at least a portion of cells included ina specimen (brightfield H&E image); applying, by the processor, to theH&E brightfield image, at least one model trained to modify pixelintensities of the H&E brightfield image so as to generate an artificialIHC image of the cells based on the H&E brightfield image, theartificial IHC image being indicative of whether the cells included inthe specimen are positive or negative for an IHC target molecule;generating the artificial IHC image by the trained model; and generatinga report based on the artificial IHC image of the cells.